Electronic device and control method thereof

Abstract

The present electronic device comprises: a memory for storing at least one instruction and a neural network model for improving an image quality; and at least one processor for executing the at least one instruction stored in the memory so as to acquire a data set for training the neural network model and train the neural network model by using the data set, wherein the at least one processor extracts a patch having a preconfigured size for each data set, obtains feature information for the extracted patch, and on the basis of the obtained feature information, determines whether to perform training for the patch in a generative adversarial network (GAN) scheme or in a non-GAN scheme, so as to train the neural network model.

Description

Electronic device and method of controlling the same

[0001] Embodiments of the present disclosure relate to an electronic device and a control method thereof, and more particularly, to an electronic device capable of performing image quality processing and a control method thereof.

[0002] Electronic devices perform operations such as generating and displaying images or providing images to a display device that displays the images. These electronic devices can display not only specific content selected by the user, as described above, but also various types of information.

[0003] Images used by electronic devices may have various resolutions, and electronic devices can display images with a lower resolution than the resolution they can display by performing upscaling technology or image quality improvement operations.

[0004] According to one embodiment of the present disclosure, an electronic device includes a memory storing at least one instruction and a neural network model for improving image quality, and at least one processor for executing at least one instruction stored in the memory to obtain a data set for training the neural network model and training the neural network model using the data set, wherein the at least one processor extracts a patch having a preset size for each of the data sets, calculates feature information for the extracted patch, and determines whether to train the patch using a generative adversarial network (GAN) method or a non-GAN method based on the calculated feature information, thereby training the neural network model.

[0005] The at least one processor may train the neural network model for the patch in a GAN manner if the produced feature information satisfies a preset condition, and may train the neural network model for the patch in a non-GAN manner if the produced feature information does not satisfy the preset condition.

[0006] The above GAN method performs a GAN comparison operation to improve a blur area within an image using GAN loss, and the non-GAN method may be a method in which GAN loss is not applied in the loss function of the GAN method.

[0007] The above feature information is information for detecting at least one of complex parts such as a human face and text within an image, and the at least one processor may perform learning on the patch in a non-GAN manner if at least one of complex parts such as a human face or text is detected in the extracted patch, and may perform learning on the patch in a GAN manner if no complex parts such as a human face and text are detected in the extracted patch.

[0008] The at least one processor can produce statistical values ​​related to texture features for the extracted patch as feature information.

[0009] The at least one processor can generate an entropy map indicating whether each pixel in the extracted patch is similar to surrounding pixels, and can use the generated entropy map to calculate the feature information.

[0010] The at least one processor can calculate a GLCM (Glay-level co-occurrence matrix) value for each pixel in the extracted patch, and calculate at least one feature information using the calculated GLCM value.

[0011] The at least one processor may use the GLCM value to produce at least one of a contrast value indicating a contrast difference within a patch, a correlation value indicating a correlation state between pixels within a patch, a square mean value of GLCM values ​​within a patch, and a homogeneity value indicating a similarity state of GLCM values ​​with respect to a diagonal direction within a patch as the feature information.

[0012] The at least one processor may produce a plurality of feature information, apply a weight corresponding to the feature information to the produced feature information to produce a feature value, and compare the produced feature value with a preset reference value to determine whether to perform learning for the patch using a generative adversarial network (GAN) method or a non-GAN method, thereby training the neural network model.

[0013] The above neural network model may be a neural network model that generates an output image of a second resolution that is higher quality than the first resolution from an input image of a first resolution.

[0014] The at least one processor may input the selected image into a trained neural network model when an image is selected to obtain an output image.

[0015] Meanwhile, a method for controlling an electronic device according to an embodiment of the present disclosure includes a step of acquiring a data set for training a neural network model that improves image quality, and a step of training the neural network model using the data set, wherein the training step includes a step of extracting a patch having a preset size for each of the data sets, a step of calculating feature information for the extracted patch, and a step of determining whether to perform training for the patch using a generative adversarial network (GAN) method or a non-GAN method based on the calculated feature information, thereby training the neural network model.

[0016] The step of calculating the above feature information can calculate statistical values ​​related to texture features for the extracted patch as the feature information.

[0017] The step of calculating the above feature information may include generating an entropy map indicating whether each pixel in the extracted patch is similar to surrounding pixels, and calculating the feature information using the generated entropy map.

[0018] The step of calculating the above feature information may calculate a GLCM (Glay-level co-occurrence matrix) value for each pixel in the extracted patch, and calculate at least one feature information using the calculated GLCM value.

[0019] The step of calculating the above feature information may use the GLCM value to calculate at least one of a contrast value indicating a contrast difference within a patch, a correlation value indicating a correlation state between pixels within a patch, a square mean value of GLCM values ​​within a patch, and a homogeneous value indicating a similar state of GLCM values ​​with respect to a diagonal direction within a patch as the feature information.

[0020] The step of calculating the above feature information calculates a plurality of feature information, calculates a feature value by applying a weight corresponding to the feature information to the calculated feature information, and the step of training the neural network model compares the calculated feature value with a preset reference value to determine whether to proceed with learning for the patch using a GAN (Generative adversarial network) method or a non-GAN method, thereby allowing the neural network model to be trained.

[0021] The above feature information is information for detecting at least one of complex parts such as a human face or text within an image, and the step of training the neural network model may include: if at least one of complex parts such as a human face or text is detected in the extracted patch, training for the patch may be performed in a non-GAN manner; and if complex parts such as a human face and text are not detected in the extracted patch, training for the patch may be performed in a GAN manner.

[0022] The present control method may further include a step of inputting the selected image into a learned neural network model to obtain an output image when an image is selected.

[0023] In a non-transitory computer-readable recording medium storing a program for executing a control method of an electronic device according to an embodiment of the present disclosure, the control method includes a step of acquiring a data set for training a neural network model for improving image quality, and a step of training the neural network model using the data set, wherein the training step includes a step of extracting a patch having a preset size for each of the data sets, a step of calculating feature information for the extracted patch, and a step of determining whether to perform training for the patch using a generative adversarial network (GAN) method or a non-GAN method based on the calculated feature information, thereby training the neural network model.

[0024] The above-described and other aspects, features, and advantages of the embodiments of the present disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings. In the accompanying drawings:

[0025] FIG. 1 is a diagram illustrating a picture quality improvement system using a neural network model according to an embodiment of the present disclosure;

[0026] FIG. 2 is a diagram illustrating a configuration of an electronic device according to an embodiment of the present disclosure;

[0027] FIG. 3 is a diagram illustrating a configuration of an electronic device according to an embodiment of the present disclosure;

[0028] Figure 4 is a diagram showing an example of image quality degradation due to application of GAN.

[0029] FIG. 5 is a diagram for explaining the selective GAN application operation of the present disclosure;

[0030] FIG. 6 is a drawing for explaining an example of generating feature information according to an embodiment of the present disclosure;

[0031] FIG. 7 is a drawing for explaining an example of generating feature information according to an embodiment of the present disclosure;

[0032] FIG. 8 is a drawing for explaining an example of generating feature information according to an embodiment of the present disclosure;

[0033] FIG. 9 is a flowchart for explaining a control method according to an embodiment of the present disclosure, and

[0034] FIG. 10 is a flowchart illustrating a selective learning method according to one embodiment of the present disclosure.

[0035] The present embodiments may be modified and have various embodiments. Specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the scope to specific embodiments, but should be understood to encompass various modifications, equivalents, and / or alternatives of the embodiments of the present disclosure. In connection with the description of the drawings, similar reference numerals may be used for similar components.

[0036] In describing the present disclosure, if it is determined that a specific description of a related known function or configuration may unnecessarily obscure the gist of the present disclosure, a detailed description thereof will be omitted.

[0037] Additionally, the following embodiments may be modified in various other forms, and the scope of the technical concepts of the present disclosure is not limited to the following embodiments. Rather, these embodiments are provided to further faithfully and completely convey the technical concepts of the present disclosure to those skilled in the art.

[0038] The terminology used in this disclosure is for the purpose of describing specific embodiments only and is not intended to limit the scope of the rights. Singular expressions include plural expressions unless the context clearly dictates otherwise.

[0039] In this disclosure, expressions such as “has,” “can have,” “includes,” or “may include” indicate the presence of a corresponding feature (e.g., a component such as a number, function, operation, or part), and do not exclude the presence of additional features.

[0040] In this disclosure, expressions such as “A or B,” “at least one of A and / or B,” or “one or more of A or / and B” can include all possible combinations of the listed items. For example, “A or B,” “at least one of A and B,” or “at least one of A or B” can all refer to (1) including at least one A, (2) including at least one B, or (3) including both at least one A and at least one B.

[0041] The expressions “first,” “second,” “first,” or “second,” etc., used in this disclosure can describe various components, regardless of order and / or importance, and are only used to distinguish one component from another, but do not limit the components.

[0042] When it is said that a component (e.g., a first component) is “(operatively or communicatively) coupled with / to” or “connected to” another component (e.g., a second component), it should be understood that said component may be directly coupled to said other component, or may be coupled via another component (e.g., a third component).

[0043] On the other hand, when it is said that a component (e.g., a first component) is "directly connected" or "directly connected" to another component (e.g., a second component), it can be understood that no other component (e.g., a third component) exists between said component and said other component.

[0044] The expression "configured to" as used in the present disclosure may be used interchangeably with, for example, "suitable for," "having the capacity to," "designed to," "adapted to," "made to," or "capable of." The term "configured to" may not necessarily mean only "specifically designed to" in terms of hardware.

[0045] Instead, in some contexts, the phrase "a device configured to" may mean that the device, in conjunction with other devices or components, is "capable of" performing A, B, and C. For example, the phrase "a processor configured (or set) to perform A, B, and C" may refer to a dedicated processor (e.g., an embedded processor) for performing those operations, or a general-purpose processor (e.g., a CPU or application processor) that can perform those operations by executing one or more software programs stored in a memory device.

[0046] In the embodiments, a 'module' or 'part' performs at least one function or operation, and may be implemented as hardware or software, or as a combination of hardware and software. Furthermore, a plurality of 'modules' or 'parts' may be integrated into at least one module and implemented as at least one processor, except for a 'module' or 'part' that needs to be implemented as a specific hardware.

[0047] According to various embodiments, operations performed by a module, program or other component may be executed sequentially, in parallel, iteratively or heuristically, or at least some operations may be executed in a different order, omitted, or other operations may be added.

[0048] Meanwhile, the various elements and areas in the drawings are schematically drawn. Therefore, the technical concept of the present invention is not limited by the relative sizes or spacing depicted in the attached drawings.

[0049] Meanwhile, an electronic device according to various embodiments of the present disclosure may include, for example, at least one of a smart phone, a tablet PC, a desktop PC, a server, a laptop PC, or a wearable device. The wearable device may include at least one of an accessory type (e.g., a watch, a ring, a bracelet, an anklet, a necklace, glasses, contact lenses, or a head-mounted device (HMD)), a fabric or clothing-integrated type (e.g., an electronic garment), a body-attached type (e.g., a skin pad or tattoo), or a bio-implantable circuit.

[0050] In some embodiments, the electronic device may include at least one of, for example, a television, a digital video disk (DVD) player, an audio device, a refrigerator, an air conditioner, a vacuum cleaner, an oven, a microwave oven, a washing machine, an air purifier, a set-top box, a home automation control panel, a security control panel, a media box (e.g., Samsung HomeSync™, Apple TV™, or Google TV™), a game console (e.g., Xbox™, PlayStation™), an electronic dictionary, an electronic key, a camcorder, or an electronic picture frame. Meanwhile, among the above-described electronic devices, a device having a display may be referred to as a display device. Meanwhile, the electronic device of the present disclosure may be a set-top box or a PC that provides images to a display device even if it does not have a display.

[0051] Hereinafter, with reference to the attached drawings, embodiments according to the present disclosure will be described in detail so that a person having ordinary knowledge in the technical field to which the present disclosure pertains can easily implement the present disclosure.

[0052] FIG. 1 is a diagram illustrating a picture quality improvement system using a neural network model according to one embodiment of the present disclosure.

[0053] Referring to FIG. 1, a neural network model (200) is disclosed. The neural network model may be referred to as a neural network, a network model, or a deep learning model.

[0054] A neural network model according to the present disclosure can input an image and perform image quality enhancement processing on the input image to generate an output image. For example, the neural network model can input a low-resolution image and output a high-resolution image. In this case, the neural network model can perform an image quality enhancement operation that not only upscales the image to high resolution, but also improves artifacts within the image. Furthermore, the neural network model can perform only the image quality enhancement operation without upscaling.

[0055] Meanwhile, in the illustrated example, for the sake of ease of explanation, image quality improvement and upscaling operations are performed using only a neural network model. However, an upscaler (not shown) may be provided at the front / back end of the neural model (200) or within the model during implementation.

[0056] The learning operation for the neural network model is described later in Fig. 5. Fig. 1 only describes the process of using the neural network model generated according to the method described later.

[0057] Referring to FIG. 1, an input image can be input to a neural network model (200).

[0058] The neural network model (200) may include multiple convolutional layers (rectangular in the illustrated image) and spatial layers (trapezoidal). Each of the multiple convolutions may include multiple nodes (or weight nodes).

[0059] Although the illustrated example only shows one channel, the neural network model may be composed of multiple channels with multiple convolutional layers arranged in parallel.

[0060] When an input image is input, the neural network model (200) can decompose the input image into multiple patches, and generate an output image for each patch by passing the decomposed multiple patches through the neural network model (200). Here, a patch is a processing unit in the neural network model (200) and can have, for example, a w*h pixel size (where w is the number of horizontal pixels and h is the number of height pixels).

[0061] And the neural network model (200) can generate an output image (20) by merging output images for each patch.

[0062] The neural network model according to the present disclosure is trained using a GAN learning method for improving a blurred area and a non-GAN learning method for a preset area, so that an output image is generated in such a way that blur processing is not applied to areas with high complexity, such as faces / texts, and blur processing is applied to other areas, thereby improving a blurred area, and generating an output image without occurrence of artifacts is possible even in areas that do not require blur processing, such as faces or text areas.

[0063] FIG. 2 is a diagram illustrating a configuration of an electronic device according to an embodiment of the present disclosure.

[0064] Referring to FIG. 2, the electronic device (100) may include a memory (110) and at least one processor (120).

[0065] The memory (110) may be implemented as an internal memory such as a ROM (e.g., an electrically erasable programmable read-only memory (EEPROM)) or a RAM included in at least one processor (120), or may be implemented as a separate memory from at least one processor (120). In this case, the memory (110) may be implemented as a memory embedded in the electronic device (100) or as a memory that can be detachably attached to the electronic device (100) depending on the purpose of data storage. For example, data for driving the electronic device (100) may be stored in a memory embedded in the electronic device (100), and data for expanding functions of the electronic device (100) may be stored in a memory that can be detachably attached to the electronic device (100).

[0066] The memory (110) may include at least one instruction. The memory (110) may include an instruction for using a neural network model described below (i.e., improving image quality for an image) or an instruction for training a neural network model.

[0067] The memory (110) stores a neural network model. The neural network model according to the present disclosure includes multiple convolutional layers, and each node within each convolutional layer may be a model trained by selectively applying the GAN or non-GAN method described below to each patch. The learning operation of the neural network model is described in detail in FIG. 5.

[0068] The memory (110) can store a data set for learning a neural network model. In addition, the memory (110) can store an image to be improved in image quality or an output image of the neural network model.

[0069] Meanwhile, in the case of memory embedded in the electronic device (100), it may be implemented as at least one of volatile memory (e.g., dynamic RAM (DRAM), static RAM (SRAM), or synchronous dynamic RAM (SDRAM)), non-volatile memory (e.g., one time programmable ROM (OTPROM), programmable ROM (PROM), erasable and programmable ROM (EPROM), electrically erasable and programmable ROM (EEPROM), mask ROM, flash ROM, flash memory (e.g., NAND flash or NOR flash), hard drive, or solid state drive (SSD)), and in the case of memory that can be detachably attached to the electronic device (100), it may be implemented as a memory card (e.g., compact flash (CF), secure digital (SD), micro secure digital (Micro-SD), mini secure digital (Mini-SD), extreme digital (xD), multi-media card (MMC), etc.), external memory that can be connected to a USB port (e.g., USB memory), etc. there is.

[0070] Meanwhile, in the illustrated example, the electronic device (100) is depicted as being composed of one memory, but when referring to volatile memory and non-volatile memory separately, the electronic device (100) may be referred to as including multiple memories.

[0071] At least one processor (120) can perform overall control operations of the electronic device (100). Specifically, at least one processor (120) has a function of controlling overall operations of the electronic device (100).

[0072] At least one processor (120) may be implemented as a digital signal processor (DSP), a microprocessor, a time controller (TCON) that processes a digital signal. However, the present invention is not limited thereto, and may include one or more of a central processing unit (CPU), a micro controller unit (MCU), a micro processing unit (MPU), a controller, an application processor (AP), a graphics-processing unit (GPU), a communication processor (CP), an ARM processor, or may be defined by the relevant term. In addition, at least one processor (120) may be implemented as a system on chip (SoC) or large scale integration (LSI) having a built-in processing algorithm, or may be implemented in the form of a field programmable gate array (FPGA). In addition, at least one processor (120) may perform various functions by executing computer executable instructions stored in a memory. Meanwhile, in FIG. 2, only one processor is included in the electronic device (100). Although shown as such, the implementation may include multiple processors (e.g., CPU + GPU, CPU + DSP).

[0073] At least one processor (120) may execute at least one instruction stored in memory (110) to obtain a data set for training a neural network model. Such a data set may be pre-stored in memory (110).

[0074] At least one processor (120) can train a neural network model using a data set. For example, at least one processor (120) can extract patches having a preset size for each data set. For example, multiple patches having a preset size (e.g., w*h size in pixels) at various locations can be extracted for a single image.

[0075] The preset size here can correspond to the unit size processed by each convolutional layer of the neural network model. Furthermore, the location from which the patch is extracted can be random, and the extraction location can also be based on the learning results of the neural network model.

[0076] At least one processor (120) generates feature information for the extracted patch. For example, at least one processor (120) may generate statistical values ​​related to texture features, extract an entropy map, generate GLCM values, and use the generated GLCM values ​​to generate feature information. The operation of generating feature information is described below with reference to FIGS. 5 to 8.

[0077] At least one processor (120) can train a neural network model by determining whether to train the patch using a Generative Adversarial Network (GAN) method or a non-GAN method based on the generated feature information. For example, if the generated feature information satisfies a preset condition, the at least one processor (120) can train a neural network model using a GAN method for the patch, and if the generated feature information does not satisfies the preset condition, the at least one processor (120) can train a neural network model using a non-GAN method for the patch.

[0078] To rephrase the above-described operation, the extraction of the above-described feature information is intended to determine whether the patch corresponds to a region with high complexity (e.g., a face region, a text region) or a region where blurring is not desirable. Accordingly, if the extracted feature information indicates a region with high complexity or a region where blurring is not desirable, at least one processor (120) can train the neural network model in a non-GAN manner. Conversely, if the current patch corresponds to a region with low complexity and where blurring is desirable, at least one processor (120) can train the neural network model in a GAN manner.

[0079] Here, the GAN method is a learning method that uses the GAN loss to perform GAN comparison operations to improve block regions within an image. The non-GAN method is a learning method that does not apply the GAN loss described above in the GAN method.

[0080] Meanwhile, while the above description describes the learning operation as being performed in two ways: with or without the optional application of GAN loss, the implementation may involve multiplying the GAN loss by a preset feature value. For example, context values ​​(or feature values) may be applied to the GAN loss, varying the degree of GAN loss application. If implemented in this way, the processor's operation can be expressed as follows.

[0081] At least one processor (120) can be said to train a neural network model by changing the ratio of GAN loss based on the produced feature values.

[0082] As described above, the electronic device (100) according to the present embodiment selectively performs GAN learning or non-GAN learning according to feature values, or performs learning by varying the GAN loss applied according to feature values, thereby preventing excessive application of blur improvement to image areas with high complexity, such as faces or text, thereby enabling image quality improvement processing.

[0083] Meanwhile, while only a simple configuration of the electronic device (100) has been illustrated and described above, various additional configurations may be provided during implementation. This will be described below with reference to FIG. 3.

[0084] FIG. 3 is a diagram illustrating a configuration of an electronic device according to an embodiment of the present disclosure.

[0085] Referring to FIG. 3, the electronic device (100') may include a memory (110), at least one processor (120), a communication device (130), a display (140), and an operating device (150).

[0086] The configuration of the memory (110) and at least one processor (120) has been described above in FIG. 2, and only operations different from those in FIG. 2 will be described below.

[0087] The communication device (130) is a configuration that performs communication with various types of external devices according to various types of communication methods. The communication device (130) may include a Wi-Fi module, a Bluetooth module, an infrared communication module, a wireless communication module, etc. Here, each communication module may be implemented in the form of at least one hardware chip.

[0088] In addition to the above-described communication method, the wireless communication module may include at least one communication chip that performs communication according to various wireless communication standards such as zigbee, 3G (3rd Generation), 3GPP (3rd Generation Partnership Project), LTE (Long Term Evolution), LTE-A (LTE Advanced), 4G (4th Generation), 5G (5th Generation), etc.

[0089] In addition, the communication device (110) may include at least one of wired communication modules that perform communication using a LAN (Local Area Network) module, an Ethernet module, a pair cable, a coaxial cable, an optical fiber cable, or a UWB (Ultra Wide-Band) module.

[0090] The communication device (130) can receive a data set. Specifically, the communication device (130) can receive a data set for training a neural network model from a server (not shown) that provides the data set.

[0091] The communication device (130) can input an image to be subjected to image enhancement or upscaling. Furthermore, the communication device (130) can output an image to which image enhancement or upscaling has been applied. Such an image may be a single still image or a video composed of multiple images. Furthermore, the image may be received as an image itself, received as an image within a webpage, or may be an image composed from a broadcast signal. For example, if the electronic device according to the present disclosure is a TV, it may receive a low-resolution broadcast signal, generate an image corresponding to the received broadcast signal, and generate an image by upscaling the generated image using the method described above (i.e., applying it to a neural network model).

[0092] Alternatively, if the electronic device according to the present disclosure is a laptop, smartphone, tablet, or the like that displays a web page, it is possible to receive a web page including a low-resolution image, and, in the process of displaying the received web page, upscale the included low-resolution image to a high-resolution image, and display the web page including the upscaled high-resolution image.

[0093] The communication device (130) can transmit and receive the learned neural network model. For example, if the electronic device (100) is a server that trains a neural network model, the communication device (130) can be controlled to transmit the neural network model on which the training operation has been performed to another device so that the other device can utilize it. Meanwhile, if the electronic device (100) is a TV or the like, the communication device (130) can receive a learned neural network model provided by the manufacturer.

[0094] The display (140) may be implemented as a variety of displays such as a Liquid Crystal Display (LCD), an Organic Light Emitting Diodes (OLED) display, a Plasma Display Panel (PDP), a projector, etc. The display (140) may also include a driving circuit, a backlight unit, etc., which may be implemented as a form such as an a-si TFT, an LTPS (low temperature poly silicon) TFT, an OTFT (organic TFT), etc. Meanwhile, the display (140) may be implemented as a touch screen combined with a touch sensor, a flexible display, a 3D display, etc.

[0095] The display (140) can display various images. For example, the display (140) can display a low-resolution image input through a communication device, or an image for which image quality improvement processing has been performed.

[0096] The operating device (150) can receive various settings for the electronic device. Such an operating device (150) can be combined with the above-described display (140) to be implemented as a single device (e.g., a touch screen).

[0097] The operating device (150) can set a method of image quality improvement processing. For example, the operating device (150) can receive input from the user whether to perform image quality improvement processing on an input image, or if so, the method of image quality improvement. For example, the operating device (150) can receive input from the user on whether to apply image quality improvement processing on an input image, whether to apply an upscaling method when applying image quality improvement processing, whether to apply blur improvement processing, and, furthermore, whether to not apply blur improvement processing to a feature region when applying blur improvement processing.

[0098] In addition, the operating device (150) can receive inputs such as the method of the learning process of the neural network model. For example, the data set to be applied to the neural network model can be selected, or the learning method for the neural network model (e.g., applying only the GAN method or a combination of the GAN method and the non-GAN method as in the present disclosure) can be received. In addition, the operating device (150) can also receive inputs such as the combination of feature values ​​to be set, which will be described later.

[0099] At least one processor (120) may select a data set to be applied when a learning command for a neural network model (200) stored in memory is input. Once the data set is selected, at least one processor (120) may control a communication device (130) to acquire the data set.

[0100] Once a data set is acquired, at least one processor (120) can perform a learning operation on a neural network model using the data set according to a preset learning method or a learning method selected by the user. Once learning of the neural network model is completed, image processing using the learned neural network model can also be performed.

[0101] Meanwhile, the electronic device (100) may receive an externally learned neural network model through a communication device (130) and perform image quality processing on an input image using the received neural network model.

[0102] Meanwhile, although the electronic device (100) in FIG. 3 is illustrated as including a display, an operating device, etc., if the electronic device (100) is a device that only trains a neural network model, the display, the operating device, etc. may be omitted. In addition, if the electronic device (100) is a device that utilizes a trained neural network model, it may further include other, not-illustrated, components, such as a speaker and a microphone, in addition to the illustrated components.

[0103] Figure 4 is a diagram illustrating an example of image quality degradation due to application of GAN.

[0104] Referring to Figure 4, the text area within the output image is shown using a neural network model trained using the GAN method for improving the blur area.

[0105] Specifically, the GAN method according to the present disclosure is a learning method applied to improve blur areas during the training process of a neural network model that performs image quality enhancement processing. For example, the GAN method trains two networks adversarially: one network generates data similar to real data, and the other model determines the generated results.

[0106] In this disclosure, the GAN method is applied to improve blurry areas within an image, which has the effect of improving blurry areas throughout the image. However, as illustrated in FIG. 4, when the GAN method is applied to images such as text to improve blur, the blur improvement may be excessive, creating artifacts. As illustrated, in the case of text, excessive blur improvement may cause the borders of the text to deteriorate, as illustrated.

[0107] Also, although not shown in the city, blur enhancement can actually create an effect when applied to areas with high texture, such as human faces or text.

[0108] To improve this point, the present disclosure applies a method to prevent blur enhancement from being applied or to prevent blur enhancement from being applied excessively to areas with high texture complexity, such as text areas and face areas.

[0109] Meanwhile, for the above-described operation, a method is required to determine whether the area in question is a region with high texture complexity, such as a text area or a face area. If the area is determined to have high texture complexity, consideration must be given to how to operate differently from existing GAN methods. The operation of the present disclosure will be described in more detail below with reference to FIGS. 5 through 8.

[0110] FIG. 5 is a diagram for explaining the selective GAN application operation of the present disclosure.

[0111] Referring to FIG. 5, a neural network model (200) is disclosed. The neural network model may be referred to as a neural network, a network model, or a deep learning model. When an image is input, the neural network model (200) can output an image with improved image quality. Furthermore, when a low-resolution image is input, the neural network model can output a high-resolution image with a higher resolution than the input image.

[0112] When simply upscaling low-quality images, the lack of pixel information can cause artifacts to become noticeable, so recently, upscaling is performed using a neural network model (200). Upscaling using a neural network model can fill in the missing pixel information based on learned information, resulting in higher-quality results.

[0113] However, because the neural network model performs an upscaling operation using learned network values ​​to compensate for the lack of pixel information, the output image can sometimes be somewhat blurry. To improve this blur, previously applied generative adversarial networks (GANs) during the neural network model training process, achieving higher-quality results than when using the neural network model alone.

[0114] For example, as illustrated, a judgment model (S510) is used to compare the output image (20) of the neural network model with a high-quality image (30) for an actual input image (10), and an operation is performed to compare whether the output image (20) and the actual high-quality image are the same or not. If it is determined that they are not the same, learning can proceed until they are determined to be the same.

[0115] However, apart from being able to obtain high-quality results for the entire image, it was confirmed that there was a problem in which the blur improvement operation was excessively processed in areas such as text areas and face areas when GAN was applied, resulting in artifacts.

[0116] In other words, GAN should be applied to improve blurred areas, but the GAN method should not be applied to areas with complex textures, such as text areas and face areas, or the GAN method should be applied to a lesser extent than other areas.

[0117] For this application, two issues must be addressed: how to determine areas where the GAN method is not applicable, or is applied to a low degree, and how to apply it to areas where it is not applicable, or is applied to a low degree, to train.

[0118] In the present disclosure, the first part is solved by extracting feature information (S520) from an image and utilizing the extracted information. Here, the feature information refers to a feature related to the texture within the image, and examples of how to derive it through roughly three methods are described below. However, the present disclosure is not limited to these methods, and if it relates to the texture characteristics of the image, methods not described in the present disclosure or methods of the present disclosure with partial modifications can be utilized.

[0119] For the second part, the above-described feature information was numerically calculated and the calculated value was applied as a correction value (or weight) for the GAN loss. In the following, the calculated value is simply multiplied by the GAN loss and used, that is, the degree of application is varied depending on the degree of texture. However, when implementing, it can also be implemented in a form where a value of 1 is output so that GAN is applied if the calculated feature value is less than a certain value, and a value of 0 is output so that GAN loss is not applied if the calculated feature value is greater than a certain value.

[0120] That is, as in step 530 shown, if the output context value (i.e., feature value) is greater than a certain value, GAN is applied, and if the output context value is less than the certain value, GAN is not applied. In addition, the above-described operation can also be implemented in an opposite form depending on the properties of the context value. That is, if the context value is proportional to the complexity, it is also possible to not apply GAN if the context value is greater than a preset value, and to apply GAN if the context value is less than the preset value.

[0121] Furthermore, while the illustrated example compares a single preset value with the calculated value, two comparison values ​​can be used during implementation. For example, if the calculated value is lower than the first value, GAN is applied. If the calculated value is between the first and second values, GAN is applied with the GAN loss adjusted proportionally to the calculated value. If the calculated value is higher than the second value, GAN is applied.

[0122] Meanwhile, the above illustrates and explains the application of the GAN method to improve the blur area of ​​the output image of the neural network model, and optionally not applying the GAN method. However, conversely, the application of the GAN method can also be implemented in a form where the GAN method is applied to improve the sharpness area, and optionally not applying the GAN method. Furthermore, while blur / sharpness was mentioned as a factor in improving image quality, the above-described method can also be used for various image quality improvement processes such as smoothing and brightness correction in addition to the factors.

[0123] Below, the learning operation of the neural network model according to the present disclosure is described based on the basic contents described above.

[0124] Neural network models can be trained using multiple data sets. First, we explain the learning process using the GAN method.

[0125] The GAN method inputs a low-resolution image from a data set into a neural network model to generate an output high-resolution image, and a comparator (510) compares the generated high-resolution output image with the actual high-resolution image to determine whether they are the same or not.

[0126] If they are not identical, each neuron of the neural network model is trained so that the two images have the same shape, and the neural network model is trained by repeating the above-described process.

[0127] In this case, the loss function of GAN can be as shown in mathematical expression 1 below.

[0128] [Mathematical Formula 1]

[0129]

[0130] As explained above, if a neural network model is trained using only the above-described loss function, artifacts may occur in areas with high complexity. Therefore, the following loss function is used in the present disclosure so that the GAN loss is not applied to areas with high complexity, or is applied with a low GAN loss.

[0131] [Equation 2]

[0132]

[0133] Here, Loss is the premise loss function, L1 is the loss function of the neural network model, is the GAN loss, and Context is a feature value of the present disclosure for optionally applying the GAN method.

[0134] Specifically, during the learning process of an image, multiple patches are extracted, and learning operations are performed in units of multiple patches. The application of the GAN method or non-GAN method of the present disclosure can be performed in units of these patches.

[0135] Accordingly, the electronic device of the present disclosure can extract a patch, generate a feature value from the extracted patch, and compare the generated feature value with a preset value to determine whether to perform learning on the patch in a GAN manner or in a non-GAN manner.

[0136] Here, the feature value is a value indicating whether blur processing is applied to text, faces, etc. within the patch, or a value numerically indicating the complexity of the texture, and can be referred to as a context value, texture value, etc.

[0137] Below, various methods for calculating feature values ​​from a patch are described with reference to FIGS. 6 to 8.

[0138] FIG. 6 is a diagram for explaining an example of generating feature information according to one embodiment of the present disclosure.

[0139] Referring to FIG. 6, multiple patches (620, 630) within an image (610) can be extracted. In the illustrated example, only a patch (620) with high complexity and a patch (530) with relatively low complexity are illustrated for explanation purposes. However, upon implementation, numerous patches can be extracted according to a preset method or the operation of a neural network model.

[0140] Each extracted patch may have different characteristics. For example, the first patch (620) may have a lot of texture. The second patch (630) may have little texture.

[0141] As described above, if blur processing is excessively applied to an image area (or patch) (620) with a lot of texture, artifacts may be created. Conversely, blur processing may improve the image quality of an image area (or patch) (630) with little texture.

[0142] As such, the GAN method needs to be applied based on the texture status of the patch. Therefore, to determine this texture status, it is necessary to quantify the texture status.

[0143] First, statistical values ​​related to texture features can be calculated and utilized. These statistical values ​​can be calculated using the following mathematical expression 3.

[0144] [Equation 3]

[0145]

[0146] Here, entropy is a statistical value related to texture features.

[0147] In the illustrated example, it can be confirmed that an area (620) with a complex texture produces a relatively high entropy value (e.g., 6.9749), and an area (630) with a non-complex texture produces a relatively low entropy value (e.g., 5.4652). In this way, it can be confirmed that the entropy value used in the present disclosure produces a relative value with respect to the complexity of the texture.

[0148] Second, values ​​related to texture features can be utilized through image transformation. Specifically, for each pixel within the extracted patch, an entropy map indicating the similarity with surrounding pixels can be generated, and feature information can be derived using this generated entropy map. For example, for each pixel within the patch, an entropy map can be derived using the brightness difference between each pixel and its nine adjacent neighboring pixels, and this derived entropy map can be utilized.

[0149] In the illustrated example, it can be seen that the entropy map (640) for the area (620) with a complex texture has a relatively white color, whereas the entropy map (650) for the area (630) with a non-complex texture has a dark color. When implemented, the entropy map can be used as feature information as is, or the average brightness value of the entropy map can be calculated and used as a feature value.

[0150] Additionally, in implementation, a GLCM matrix may be used in addition to the above-described method. This will be described below with reference to Figure 7.

[0151] FIG. 7 is a diagram illustrating an example of generating feature information according to one embodiment of the present disclosure.

[0152] Specifically, FIG. 7 is a diagram for explaining an embodiment of generating feature information using GLCM (Glay-level co-occurrence matrix) values.

[0153] GLCM, also known as intensity co-occurrence matrix or spatial intensity dependence matrix, is a statistical method for retrieving texture by considering the spatial relationships of pixels. The GLCM function calculates how frequently pairs of pixels with specific values ​​occur in a specific spatial relationship in an image, generating a GLCM, and extracting statistical measures (e.g., contrast, correlation, energy, and homogeneity) from this matrix to identify the texture characteristics of an image.

[0154] For example, if the gray value is expressed in three steps from 0 to 3, the first value of the GLCM matrix is ​​the number of adjacent two pixels having the value 00, and the second value is the number of adjacent two pixels having the value 01. A GLCM matrix can be created in this way.

[0155] And using the GLCM matrix generated in this way, the following various texture-related measurement values ​​can be calculated.

[0156] Here, the contrast value can measure local changes in the luminance co-occurrence matrix. For example, it represents the contrast difference within a patch and can be calculated using the following mathematical formula (4).

[0157] [Equation 4]

[0158]

[0159] Here, i and j are the matrix coordinates of the GLCM matrix, and these contrast values ​​have high values ​​when the texture is complex.

[0160] The correlation value represents the joint probability of a given pair of pixels, for example, it represents the correlation between pixels within a patch, and can be calculated using the following mathematical expression 5.

[0161] [Equation 5]

[0162]

[0163] These correlation values ​​can have low values ​​if the texture is complex.

[0164] The energy value represents the sum of the squares of the GLCM elements, and can be referred to as the mean square value, uniformity, and ASM value, and can be calculated using the following mathematical expression 6.

[0165] [Equation 6]

[0166]

[0167] These energy values ​​can be low if the texture is complex.

[0168] The identity value (or homogeneity) is a value that measures how closely the GLCM elements are distributed along the GLCM diagonal, indicating the similarity of the GLCM values ​​in the diagonal direction within the patch, and can be calculated using the following mathematical expression 7.

[0169] [Equation 7]

[0170]

[0171] These same values ​​may have lower values ​​if the texture is complex.

[0172] When a patch is extracted in this way, a GLCM matrix for the extracted patch is calculated, and the four feature values ​​described above (e.g., contrast value, correlation value, energy value, and identity value) can be calculated using the calculated GLCM matrix.

[0173] Meanwhile, while the above only describes a method for calculating individual feature values ​​for each patch, the above-described methods can also be combined to calculate feature values ​​during implementation. This will be described below with reference to Figure 8.

[0174] FIG. 8 is a diagram for explaining an example of generating feature information according to one embodiment of the present disclosure.

[0175] Referring to Fig. 8, this is a diagram illustrating an example of feature information calculated for two patches having different texture characteristics. Specifically, for each patch, an entropy value is calculated individually, a GLCM matrix is ​​calculated for each patch, and four feature values ​​as in Fig. 7 are calculated. The calculated entropy value and four features related to the GLCM are combined to finally calculate the feature values.

[0176] [Equation 8]

[0177]

[0178] Here, Context value is a feature value, entropy is an entropy value calculated in Fig. 5, contrast is a contrast value calculated in Fig. 7, correlation is a correlation value calculated in Fig. 7, energy is a root mean value calculated in Fig. 7, and homogeneity is a homogeneity value calculated in Fig. 7.

[0179] As explained above, the correlation value has been processed to change the sign in that the root mean square value and the homogeneity value have lower values ​​when the texture is complex, and the entropy value and contrast value have opposite values.

[0180] While the illustrated example only applies a weight of 1 / 8 to entropy, various weighting methods not described herein can be applied during implementation. Furthermore, implementations can also omit some of the aforementioned feature information.

[0181] Referring to Fig. 8, it can be confirmed that a relatively high feature value (0.7021) is produced for a patch with high texture (810), and a relatively low feature value (0.45) is produced for a patch with low texture (840).

[0182] In this case, learning can be performed in a non-GAN manner by comparing the preset value (e.g., 0.5) with the calculated feature value, and if the calculated feature value is greater than 0.5, learning can be performed by not applying GAN, and if the calculated feature value is less than 0.5, learning can be performed by applying GAN.

[0183] FIG. 9 is a drawing for explaining a control method according to one embodiment of the present disclosure.

[0184] Referring to Fig. 9, a data set for training a neural network model that improves image quality is acquired (S910).

[0185] Then, a neural network model is trained using the data set (S920). Specifically, for each data set, patches having a preset size are extracted, feature information for the extracted patches is calculated, and based on the calculated feature information, whether to train the patches using a Generative Adversarial Network (GAN) method or a non-GAN method is determined, thereby training the neural network model.

[0186] For example, for the extracted patch, statistical values ​​related to texture features can be calculated as feature information, or an entropy map indicating whether each pixel in the extracted patch is similar to surrounding pixels can be generated, and feature information can be calculated using the generated entropy map, or a GLCM (Glay-level co-occurrence matrix) value can be calculated for each pixel in the extracted patch, and at least one feature information can be calculated using the calculated GLCM value.

[0187] Meanwhile, during implementation, the aforementioned feature information can be utilized in a complex manner. In this case, individual weights can be applied to each feature information to ultimately derive the feature value.

[0188] Once the feature values ​​are calculated in this way, the feature values ​​can be compared with preset reference values ​​to determine whether to train the patch using the Generative adversarial network (GAN) method or the non-GAN method, thereby training the neural network model.

[0189] FIG. 10 is a flowchart illustrating a selective learning method according to one embodiment of the present disclosure.

[0190] Referring to Fig. 10, multiple patches can be extracted for each input data set (S1010). Each of the multiple patches may have a preset size, and the number and location of the generated patches may be based on a preset algorithm or design method of a neural network model.

[0191] When multiple patches are extracted from a single data set in this way, feature information can be derived for each patch (S1020). Specifically, as described above, entropy values ​​for patches, entropy maps, or GLCM matrices can be derived, and contrast values, correlation values, energy values, and homogeneity values ​​can be derived using the derived GLCM matrices. During implementation, only one of the derived values ​​can be used, and feature values ​​can be derived by combining the above-described values.

[0192] Once the feature value is calculated, the feature value is compared with the preset value (S1030), and if the preset condition is satisfied, the neural network can be trained in a non-GAN manner (S1040). For example, if the feature value is greater than the preset value, the neural network can be trained in a non-GAN manner. Meanwhile, in this case, the case where the feature value is greater than the preset value is considered to satisfy the preset condition, because the calculated feature value is calculated to be proportional to the complexity of the texture. In other words, if the previously calculated feature value is inversely proportional to the complexity of the texture, the case where the feature value is less than the preset value can be considered to satisfy the preset condition.

[0193] Conversely, if the preset conditions are not satisfied, the neural network can be trained using the GAN method (S1050). Meanwhile, although the above description illustrates and describes selectively training using the GAN method or the non-GAN method depending on whether the conditions are satisfied, the implementation can also be implemented in a form where the GAN loss is always applied, and the calculated feature value is applied as the weight value of the GAN loss, so that if the texture is complex, training progresses with a low GAN loss, and if the texture is simple, training progresses with a high GAN loss.

[0194] In addition, if the above-described process is repeatedly performed for all patches of all data sets, the learning of the neural network model can proceed by repeatedly applying the process so that the overall loss value has a small value.

[0195] Meanwhile, the methods according to at least some of the various embodiments of the present disclosure described above may be implemented in the form of an application that can be installed on an existing electronic device.

[0196] Additionally, the methods according to at least some of the various embodiments of the present disclosure described above can be implemented with only a software upgrade or a hardware upgrade for an existing electronic device.

[0197] Additionally, the methods according to at least some of the various embodiments of the present disclosure described above may also be performed through an embedded server provided in an electronic device, or an external server of at least one of the electronic devices.

[0198] Meanwhile, according to one embodiment of the present disclosure, the various embodiments described above can be implemented as software including instructions stored in a machine-readable storage medium that can be read by a machine (e.g., a computer). The machine is a device that can call instructions stored from the storage medium and operate according to the called instructions, and may include an electronic device (e.g., electronic device (A)) according to the disclosed embodiments. When an instruction is executed by a processor, the processor can perform a function corresponding to the instruction directly or by using other components under the control of the processor. The instruction may include code generated or executed by a compiler or an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, the 'non-transitory storage medium' only means that it is a tangible device and does not include a signal (e.g., electromagnetic wave), and this term does not distinguish between cases where data is stored semi-permanently and cases where it is temporarily stored in the storage medium. For example, a 'non-transitory storage medium' may include a buffer in which data is temporarily stored. According to one embodiment, the method according to various embodiments disclosed in the present document may be provided as a computer program product. The computer program product may be traded between sellers and buyers as a commodity. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) through an application store (e.g., Play Store™) or directly between two user devices (e.g., smartphones).In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable app) may be temporarily stored or temporarily created in a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0199] Various embodiments of the present disclosure may be implemented as software including instructions stored in a machine-readable storage medium that can be read by a machine (e.g., a computer). The device may include an electronic device (e.g., an electronic device (100)) according to the disclosed embodiments, which is a device capable of calling instructions stored in the storage medium and operating according to the called instructions.

[0200] When the above instruction is executed by the processor, the processor may perform the function corresponding to the instruction directly or by using other components under the control of the processor. The instruction may include code generated or executed by a compiler or interpreter.

[0201] Although the preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above, and various modifications may be made by a person having ordinary skill in the art to which the present disclosure pertains without departing from the gist of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical idea or prospect of the present disclosure.

Claims

1. In electronic devices, A memory storing at least one instruction and a neural network model that improves the quality of an image; and At least one processor for executing at least one instruction stored in the memory to obtain a data set for training the neural network model and training the neural network model using the data set; At least one processor of the above, An electronic device that trains the neural network model by extracting patches having a preset size for each of the above data sets, calculating feature information for the extracted patches, and determining whether to perform learning for the patches using a generative adversarial network (GAN) method or a non-GAN method based on the calculated feature information.

2. In paragraph 1, At least one processor of the above, An electronic device that trains the neural network model for the patch in a GAN manner if the above-described generated feature information satisfies a preset condition, and trains the neural network model for the patch in a non-GAN manner if the above-described generated feature information does not satisfy the preset condition.

3. In paragraph 2, The above GAN method performs a GAN comparison operation to improve the blur area in the image using GAN loss. The above non-GAN method is an electronic device in which the GAN loss is not applied in the loss function of the GAN method.

4. In paragraph 1, The above feature information is, Information for detecting at least one of a human face and text in an image, At least one processor of the above, An electronic device that performs learning on the patch in a non-GAN manner if at least one of a human face or text is detected in the extracted patch, and performs learning on the patch in a GAN manner if neither a human face nor text is detected in the extracted patch.

5. In paragraph 1, At least one processor of the above, An electronic device that calculates statistical values ​​related to texture features of the extracted patch as feature information.

6. In paragraph 1, At least one processor of the above, An electronic device that generates an entropy map that indicates whether each pixel in the extracted patch is similar to surrounding pixels, and calculates the feature information using the generated entropy map.

7. In paragraph 1, At least one processor of the above, An electronic device that calculates a GLCM (Glay-level co-occurrence matrix) value for each pixel in the extracted patch and calculates at least one feature information using the calculated GLCM value.

8. In paragraph 7, At least one processor of the above, An electronic device that uses the GLCM values ​​to produce at least one of a contrast value indicating a contrast difference within a patch, a correlation value indicating a correlation state between pixels within a patch, a square mean value of GLCM values ​​within a patch, and a homogeneous value indicating a similar state of GLCM values ​​with respect to a diagonal direction within a patch as the feature information.

9. In paragraph 8, At least one processor of the above, Generate multiple feature information, and apply a weight corresponding to the feature information to the generated feature information to generate a feature value. An electronic device that trains the neural network model by comparing the above-described generated feature values ​​with preset reference values ​​and determining whether to proceed with learning for the patch using a generative adversarial network (GAN) method or a non-GAN method.

10. In paragraph 1, The above neural network model is, An electronic device, which is a neural network model that generates an output image of a second resolution with higher quality than the first resolution from an input image of a first resolution.

11. In paragraph 1, At least one processor of the above, An electronic device that inputs the selected image into a learned neural network model when an image is selected to obtain an output image.

12. In a method for controlling an electronic device, A step of obtaining a data set for training a neural network model that improves the image quality; and A step of training the neural network model using the above data set; The above learning steps are: A step of extracting a patch having a preset size for each of the above data sets; A step of producing feature information for the extracted patch; and A control method comprising: a step of training the neural network model by determining whether to proceed with learning for the patch in a GAN (generative adversarial network) manner or a non-GAN manner based on the generated feature information; 13. In paragraph 12, The step of producing the above characteristic information is: A control method for calculating statistical values ​​related to texture features of the above extracted patches as feature information.

14. In paragraph 12, The step of producing the above characteristic information is: A control method for generating an entropy map indicating whether each pixel in the extracted patch is similar to surrounding pixels, and calculating the feature information using the generated entropy map.

15. In a non-transitory computer-readable recording medium storing a program for executing a method of controlling an electronic device, The above control method is, A step of obtaining a data set for training a neural network model that improves the image quality; and A step of training the neural network model using the above data set; The above learning steps are: A step of extracting a patch having a preset size for each of the above data sets; A step of producing feature information for the extracted patch; and A computer-readable recording medium comprising: a step of training the neural network model by determining whether to proceed with training for the patch using a generative adversarial network (GAN) method or a non-GAN method based on the generated feature information;

Images