Electronic device and control method thereof

Abstract

This electronic device comprises: a display; a memory for storing instructions; and one or more processors including processing circuitry. The one or more processors, when the instructions are individually or collectively executed, when a voice input is received while an image is displayed through the display, input the image and the voice input into an artificial intelligence model to obtain image quality adjustment information, and adjust the image quality of the image displayed through the display on the basis of the obtained image quality adjustment information.

Description

Electronic device and method of controlling the same

[0001] The present disclosure relates to an electronic device and a control method thereof, and more particularly, to an electronic device for adjusting and displaying image quality and a control method thereof.

[0002] Advances in electronic technology have led to the development and proliferation of various types of electronic devices. In particular, display devices, used in a variety of settings, including homes, offices, and public spaces, have been continuously evolving in recent years.

[0003] Recently, technologies have been improving to control the functions of display devices based on the user's natural language commands.

[0004] An electronic device according to one or more embodiments includes a display; a memory storing instructions; and one or more processors including processing circuitry, wherein the one or more processors are configured to, when the instructions are individually or collectively executed, receive a voice input while an image is displayed on the display, input the image and the voice input into an artificial intelligence model to obtain image quality adjustment information, and adjust the image quality of the image displayed on the display based on the obtained image quality adjustment information. The artificial intelligence model may be trained to output image quality adjustment information for the input image based on the image and the voice input and an image quality adjustment term corresponding to the voice input.

[0005] The one or more processors according to one or more embodiments can update the image quality adjustment information based on at least one of context information of the electronic device or usage history information of the user when the instructions are individually or collectively executed, and adjust the image quality of an image displayed through the display based on the updated image quality adjustment information.

[0006] According to one or more embodiments, the one or more processors may, when the instructions are individually or collectively executed, input at least one of context information of the electronic device or user usage history information, the image quality adjustment information, and the feature information of the image into a regression model to obtain the updated image quality adjustment information. The regression model may be trained to output the updated image quality adjustment information through regression analysis when at least one of context information of the electronic device or user usage history information, the image quality adjustment information, and the feature information of the image are input.

[0007] The artificial intelligence model according to one or more embodiments may be trained based on user reactions received after an image whose quality has been adjusted according to the quality adjustment information is displayed through the display.

[0008] The one or more processors according to one or more embodiments, when the instructions are individually or collectively executed, compare the image and the image whose image quality has been adjusted according to the image quality adjustment information to identify at least one area in the image whose image quality has been adjusted, identify image quality change information before and after image quality adjustment for the identified at least one area, and use the voice input and the image quality change information for learning the artificial intelligence model according to the user reaction.

[0009] The artificial intelligence model according to one or more embodiments may include a first sub-neural network model that preprocesses the input voice input and outputs a preprocessed voice; a second sub-neural network model that preprocesses the input image and outputs a preprocessed image; and a common neural network model that converts the preprocessed voice input from the first sub-neural network model and the preprocessed image input from the second sub-neural network model into the same format and outputs image quality adjustment information for the input image based on the format-converted voice and the format-converted preprocessed image.

[0010] The artificial intelligence model according to one or more embodiments may be trained to identify keyword words in the format-converted speech by applying self-attention through a transformer encoder, and to obtain the image quality adjustment information based on the format-converted image and the identified keyword words by using a regression model.

[0011] The first sub-neural network model according to one or more embodiments may preprocess the speech input through at least one of tokenization, lemmatization, or stop word removal.

[0012] The second sub-neural network model according to one or more embodiments may preprocess the input image through at least one of resizing, color space conversion, or feature extraction for the input image.

[0013] In one or more embodiments, the one or more processors, when the instructions are individually or collectively executed, when the voice input is received while the image is displayed through the display, identify whether the voice input includes a preset keyword, and when the voice input is identified as including the preset keyword, input the image and the voice input into the artificial intelligence model to obtain image quality adjustment information.

[0014] The image quality adjustment information according to one or more embodiments includes an image quality adjustment item and an adjustment level, and the image quality adjustment item may include at least one of brightness adjustment information, contrast adjustment information, or color adjustment information for the image.

[0015] According to one or more embodiments, a method for controlling an electronic device includes: when a voice input is received while an image is displayed, inputting the image and the voice input into an artificial intelligence model to obtain image quality adjustment information; and adjusting the image quality of the displayed image based on the obtained image quality adjustment information. The artificial intelligence model may be trained to output image quality adjustment information for the input image based on an image quality adjustment term corresponding to the voice input and the input image, based on the image and the voice input.

[0016] According to one or more embodiments, a non-transitory computer-readable medium storing computer instructions that, when executed by a processor of an electronic device, cause the electronic device to perform an operation, the operation comprising: when a voice input is received while an image is displayed, inputting the image and the voice input into an artificial intelligence model to obtain image quality adjustment information; and adjusting the image quality of the displayed image based on the obtained image quality adjustment information. The artificial intelligence model may be trained to output image quality adjustment information for the input image based on the image and the voice input and an image quality adjustment term corresponding to the voice input.

[0017] The above and other aspects, features and advantages of specific embodiments of the present disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings.

[0018] FIG. 1 is a drawing schematically illustrating the operation of an electronic device according to one or more embodiments.

[0019] FIG. 2A is a block diagram showing the configuration of an electronic device according to one embodiment.

[0020] FIG. 2b is a block diagram specifically illustrating a configuration of an electronic device according to one or more embodiments.

[0021] FIG. 3 is a flowchart illustrating a method of controlling an electronic device according to one or more embodiments.

[0022] FIG. 4A and FIG. 4B are drawings for explaining a learning method of an artificial intelligence model according to one or more embodiments.

[0023] FIG. 5 is a diagram illustrating a learning method of an artificial intelligence model according to one or more embodiments.

[0024] FIG. 6 is a diagram illustrating a learning method of an artificial intelligence model according to one or more embodiments.

[0025] FIG. 7a is a diagram for explaining a method for obtaining image quality adjustment information according to one or more embodiments.

[0026] FIG. 7b is a diagram for explaining a method for obtaining image quality adjustment information according to one or more embodiments.

[0027] FIG. 8 is a diagram illustrating the operation of a first sub-neural network according to one or more embodiments.

[0028] FIGS. 9A and 9B are diagrams illustrating the operation of a second sub-neural network according to one or more embodiments.

[0029] FIG. 10 is a diagram illustrating the operation of a common neural network according to one or more embodiments.

[0030] FIG. 11 and FIG. 12 are drawings for explaining the operation of a regression model according to one or more embodiments.

[0031] FIG. 13 is a diagram for explaining a method for obtaining image quality adjustment information according to one or more embodiments.

[0032] FIG. 14 is a diagram for explaining a method for obtaining image quality adjustment information according to one or more embodiments.

[0033] FIG. 15 is a diagram for explaining a method for obtaining image quality adjustment information according to one or more embodiments.

[0034] The terms used in this specification will be briefly explained, and the present disclosure will be described in detail.

[0035] The terms used in the embodiments of this disclosure have been selected from widely used, current terms, taking into account the functions of this disclosure. However, these terms may vary depending on the intentions or cases of those skilled in the art, the emergence of new technologies, etc. Furthermore, in certain cases, terms may be arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the description of the relevant disclosure. Therefore, the terms used in this disclosure should not be defined simply as names of terms, but rather based on the meanings of the terms and the overall content of this disclosure.

[0036] In this specification, expressions such as “has,” “can have,” “includes,” or “may include” indicate the presence of a feature (e.g., a number, function, operation, or component such as a part), and do not exclude the presence of additional features.

[0037] In this disclosure, expressions such as “A or B,” “at least one of A and / or B,” or “one or more of A or / and B” can include all possible combinations of the listed items. For example, “A or B,” “at least one of A and B,” or “at least one of A or B” can all refer to cases where (1) only A is included, (2) only B is included, or (3) both A and B are included.

[0038] As used herein, the expressions “first,” “second,” “first,” or “second,” etc., may describe various components, regardless of order and / or importance, and are only used to distinguish one component from another, but do not limit the components.

[0039] When it is said that a component (e.g., a first component) is “operatively or communicatively coupled with / to” or “connected to” another component (e.g., a second component), it should be understood that the component may be directly coupled to the other component, or may be connected through another component (e.g., a third component).

[0040] The expression "configured to" as used in the present disclosure may be used interchangeably with, for example, "suitable for," "having the capacity to," "designed to," "adapted to," "made to," or "capable of." The term "configured to" may not necessarily mean only "specifically designed to" in terms of hardware.

[0041] In some contexts, the phrase "a device configured to" may mean that the device, in conjunction with other devices or components, is "capable of" performing A, B, and C. For example, the phrase "a processor configured (or set) to perform A, B, and C" may refer to a dedicated processor (e.g., an embedded processor) for performing those operations, or a general-purpose processor (e.g., a CPU or application processor) that can perform those operations by executing one or more software programs stored in a memory device.

[0042] Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, terms such as "comprise" or "consist of" are intended to indicate the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, but should be understood not to preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0043] In the embodiments, a "module" or "part" performs at least one function or operation and may be implemented as hardware or software, or as a combination of hardware and software. Furthermore, a plurality of "modules" or "parts" may be integrated into at least one module and implemented as at least one processor (not shown), excluding any "module" or "part" that needs to be implemented as specific hardware.

[0044] Meanwhile, the various elements and areas in the drawings are schematically drawn. Therefore, the technical concept of the present invention is not limited by the relative sizes or spacing depicted in the attached drawings.

[0045] An embodiment of the present disclosure will be described in more detail with reference to the attached drawings below.

[0046] FIG. 1 is a drawing schematically illustrating the operation of an electronic device according to one or more embodiments.

[0047] According to FIG. 1, the electronic device (100) may be implemented as a TV, but is not limited thereto, and may be implemented as various types of display devices such as a monitor, a PC, a kiosk, a tablet PC, an electronic picture frame, a mobile phone, a HMD (Head mounted Display), a NED (Near Eye Display), a LFD (Large format display), a Digital Signage, a DID (Digital Information Display), a video wall, a projector display, etc., or an image processing device (e.g., a set-top box, one connected box) that provides images to a display device.

[0048] According to one embodiment, when a voice input for adjusting the image quality (hereinafter, a “image quality adjustment command”) is received from a user while an image (10) is displayed on a screen, the electronic device (100) may adjust the image quality of the displayed image based on the image quality adjustment command and display the image (20) with the adjusted image quality. For example, the image quality adjustment command may be a natural language command (or natural language request). The natural language command may be a command for the user to control the electronic device (100) using everyday language. For example, the natural language command may include everyday language such as “Can you adjust the image quality to a comfortable level according to the lighting in the room?”

[0049] For example, the electronic device (100) can perform image quality adjustment based on technologies such as artificial intelligence (AI), natural language processing (NLP), computer vision, and deep learning. Artificial intelligence is a technology that enables machines to think and judge like humans, and can understand the user's intention through natural language commands and process it appropriately. Natural language processing can convert the natural language commands entered by the user into a form that the computer can understand. For example, natural language processing can analyze text data to extract meaning and perform various tasks such as syntax analysis, named entity recognition, and sentiment analysis. Computer vision is a technology that enables the displayed machine to understand and analyze images, and can analyze characteristics of the image such as color, brightness, and contrast to perform appropriate image quality adjustment. Deep learning is a technology that uses artificial neural networks to learn large amounts of data and identify complex patterns based on the data, and can be used for more precise natural language processing and computer vision processing.

[0050] We will explain various examples of implementing image quality adjustment services through natural language commands using technologies such as artificial intelligence, natural language processing, computer vision, and deep learning.

[0051] FIG. 2A is a block diagram showing the configuration of an electronic device according to one embodiment.

[0052] According to FIG. 2a, the electronic device (100) includes a display (110), a memory (120), and one or more processors (130).

[0053] The display (110) may be implemented as a display including a self-luminous element or a display including a non-luminous element and a backlight. For example, it may be implemented as various types of displays such as an LCD (Liquid Crystal Display), an OLED (Organic Light Emitting Diodes) display, an LED (Light Emitting Diodes), a micro LED, a Mini LED, a PDP (Plasma Display Panel), a QD (Quantum dot) display, a QLED (Quantum dot light-emitting diodes), etc. The display (110) may also include a driving circuit, a backlight unit, etc., which may be implemented in a form such as an a-si TFT, an LTPS (low temperature poly silicon) TFT, an OTFT (organic TFT), etc. According to an example, a touch sensor that detects a touch operation in the form of a touch film, a touch sheet, a touch pad, etc. may be disposed on the front of the display (110) so as to be implemented so as to detect various types of touch inputs. For example, the display (110) can detect various types of touch inputs, such as a touch input by a user's hand, a touch input by an input device such as a stylus pen, and a touch input by a specific electrostatic material. Here, the input device can be implemented as a pen-type input device that can be referred to by various terms such as an electronic pen, a stylus pen, an S-pen, etc. According to an example, the display (110) can be implemented as a flat display, a curved display, a flexible display that can be folded or / and rolled, etc.

[0054] The memory (120) can store data required for various embodiments. The memory (120) may be implemented in the form of memory embedded in the electronic device (100') or in the form of memory that can be detachably attached to the electronic device (100) depending on the purpose of data storage. For example, data for driving the electronic device (100) may be stored in a memory embedded in the electronic device (100'), and data for expanding the functions of the electronic device (100) may be stored in a memory that can be detachably attached to the electronic device (100). Meanwhile, in the case of memory embedded in the electronic device (100), it may be implemented as at least one of volatile memory (e.g., dynamic RAM (DRAM), static RAM (SRAM), or synchronous dynamic RAM (SDRAM)), non-volatile memory (e.g., one time programmable ROM (OTPROM), programmable ROM (PROM), erasable and programmable ROM (EPROM), electrically erasable and programmable ROM (EEPROM), mask ROM, flash ROM, flash memory (e.g., NAND flash or NOR flash), hard drive, or solid state drive (SSD). In addition, in the case of memory that can be attached or detached to the electronic device (100'), it may be implemented as at least one of memory cards (e.g., compact flash (CF), secure digital (SD), micro secure digital (Micro-SD), mini secure digital (Mini-SD), extreme digital (xD), multi-media card (MMC), etc.), external memory that can be connected to a USB port (e.g., USB memory), etc. It can be implemented in the form of.

[0055] As an example, the memory (120) may store a computer program including at least one instruction or instructions for controlling the electronic device (100).

[0056] One or more processors (130) control the overall operation of the electronic device (100). Specifically, one or more processors (130) may be connected to each component of the electronic device (100) to control the overall operation of the electronic device (100). For example, one or more processors (130) may be electrically connected to the display (110) and the memory (120) to control the overall operation of the electronic device (100). One or more processors (130) may be configured as one or more processors.

[0057] One or more processors (130) may perform operations of the electronic device (100) according to various embodiments by executing at least one instruction stored in the memory (120).

[0058] In one example, the artificial intelligence related functions according to the present disclosure may be operated through the processor and memory of an electronic device.

[0059] One or more processors (130) may be composed of one or more processors. In this case, one or more processors may include at least one of a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and an NPU (Neural Processing Unit), but are not limited to the examples of the processors described above.

[0060] CPUs are general-purpose processors capable of performing not only general calculations but also artificial intelligence calculations. Their multi-layered cache structure allows for the efficient execution of complex programs. CPUs are advantageous for serial processing, enabling organic linking of previous and subsequent calculation results through sequential calculations. General-purpose processors are not limited to the examples described above, except where specifically identified as CPUs.

[0061] A GPU is a processor designed for large-scale computations, such as floating-point operations used in graphics processing. It integrates a large number of cores to perform large-scale computations in parallel. In particular, GPUs may be advantageous over CPUs in parallel processing methods, such as convolution operations. Furthermore, GPUs can be used as coprocessors to supplement the functions of CPUs. Processors for large-scale computations are not limited to the examples described above, except in cases where they are specifically referred to as GPUs.

[0062] An NPU is a processor specialized in artificial intelligence computation using artificial neural networks, and each layer of the artificial neural network can be implemented in hardware (e.g., silicon). Since NPUs are designed specifically according to the company's specifications, they have less freedom than CPUs or GPUs, but can efficiently process the AI ​​computations requested by the company. Meanwhile, as a processor specialized in AI computation, an NPU can be implemented in various forms, such as a Tensor Processing Unit (TPU), an Intelligence Processing Unit (IPU), or a Vision Processing Unit (VPU). Except as specifically designated as an NPU, an AI processor is not limited to the examples described above.

[0063] Additionally, one or more processors (130) may be implemented as a System on Chip (SoC). In this case, the SoC may further include, in addition to one or more processors (130), a memory (120), and a network interface such as a bus for data communication between the processor (130) and the memory (120).

[0064] When a plurality of processors are included in a SoC (System on Chip) included in an electronic device (100), the electronic device (100) may perform operations related to artificial intelligence (e.g., operations related to learning or inference of a neural network model) by using some of the plurality of processors. For example, the electronic device may perform operations related to artificial intelligence by using at least one of a GPU, NPU, VPU, TPU, or hardware accelerator specialized in artificial intelligence operations such as convolution operations or matrix multiplication operations among the plurality of processors. However, this is only one or more embodiments, and it is of course possible to process operations related to artificial intelligence by using a CPU or a general-purpose processor.

[0065] Additionally, the electronic device (100) can perform operations related to functions related to artificial intelligence by utilizing multiple cores (e.g., dual cores, quad cores, etc.) included in a single processor. In particular, the electronic device can perform artificial intelligence operations, such as convolution operations and matrix multiplication operations, in parallel by utilizing multiple cores included in the processor.

[0066] One or more processors (130) are controlled to process data based on predefined operating rules or artificial intelligence models (neural network models) stored in memory (120). The predefined operating rules or neural network models are characterized by being created through learning.

[0067] Here, "created through learning" means that a predefined set of behavioral rules or a neural network model with desired characteristics is created by applying a learning algorithm to a large number of learning data. This learning may be performed on the device itself, where the artificial intelligence according to the present disclosure is implemented, or through a separate server / system.

[0068] A neural network model may be composed of multiple neural network layers. At least one layer has at least one weight value and performs its operation through the operation result of the previous layer and at least one defined operation. Examples of neural networks include a convolutional neural network (CNN), a recurrent neural network (RNN), a deep neural network (DNN), a restricted boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), deep Q-networks, and a transformer. The neural networks in the present disclosure are not limited to the above-described examples unless otherwise specified.

[0069] A learning algorithm is a method for training a target device (e.g., a robot) using a plurality of learning data sets so that the target device can make decisions or predictions on its own. Examples of learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning. The learning algorithms in the present disclosure are not limited to the aforementioned examples unless otherwise specified. For convenience of explanation, one or more processors (130) will be referred to as "processors (130)" below.

[0070] FIG. 2b is a block diagram specifically illustrating a configuration of an electronic device according to one or more embodiments.

[0071] According to FIG. 2b, the electronic device (100') may include a display (110), a memory (120), one or more processors (130), a communication interface (140), a user interface (150), a speaker (160), and a microphone (170). Among the configurations illustrated in FIG. 2b, a detailed description of configurations that overlap with those illustrated in FIG. 2a will be omitted.

[0072] It goes without saying that the communication interface (140) can be implemented as various interfaces depending on the implementation example of the electronic device (100'). For example, the communication interface (140) can communicate with an external device, an external storage medium (e.g., a USB memory), an external server (e.g., a web hard drive), etc. through a communication method such as Bluetooth, AP-based Wi-Fi (Wireless LAN network), Zigbee, wired / wireless LAN (Local Area Network), WAN (Wide Area Network), Ethernet, IEEE 1394, HDMI (High-Definition Multimedia Interface), USB (Universal Serial Bus), MHL (Mobile High-Definition Link), AES / EBU (Audio Engineering Society / European Broadcasting Union), optical, coaxial, etc. According to one example, the communication interface (140) can communicate with another electronic device, an external server, and / or a remote control device.

[0073] The user interface (150) may be implemented as a device such as a button, a touch pad, a mouse, and a keyboard, or as a touch screen that can also perform the display function and operation input function described above.

[0074] The speaker (160) may be configured to output various audio data as well as various notification sounds or voice messages. The processor (130) may control the speaker (160) to output feedback or various notifications in audio format according to various embodiments of the present disclosure.

[0075] The microphone (170) is configured to receive user voice or other sounds and convert them into audio data. However, according to another embodiment, the electronic device (100') may receive user voice input via an external device through the communication interface (140).

[0076] In addition, the electronic device (100') may further include a camera and sensors, etc., depending on the implementation example.

[0077] The camera can be turned on and take pictures based on a preset event. The camera can convert the captured image into an electrical signal and generate image data based on the converted signal. For example, a subject can be converted into an electrical image signal through a semiconductor optical element (CCD; Charge Coupled Device), and the converted image signal can be amplified and converted into a digital signal and then signal processed. For example, the camera can include at least one of a standard (or basic) camera and an ultra-wide-angle camera.

[0078] Sensors may include various types of sensors, such as touch sensors, proximity sensors, acceleration sensors (or gravity sensors), geomagnetic sensors, gyro sensors, pressure sensors, position sensors, distance sensors, light sensors, etc.

[0079] Meanwhile, according to an implementation example of the electronic device (100'), a speaker, a tuner, and a demodulator may be additionally included. The tuner (not shown) can receive an RF (Radio Frequency) broadcast signal by tuning a channel selected by a user or all pre-stored channels among RF broadcast signals received through an antenna. The demodulator (not shown) can receive and demodulate a digital IF signal (DIF) converted by the tuner, and can also perform channel decoding, etc. According to one embodiment, an input image received through the tuner can be processed through the demodulator (not shown) and then provided to the processor (140).

[0080] FIG. 3 is a flowchart illustrating a method of controlling an electronic device according to one or more embodiments.

[0081] According to FIG. 3, in operation 310, the electronic device (100) can identify whether a voice input is received while the image is being displayed. For example, the voice input may include a natural language input (or a natural language command).

[0082] For example, the electronic device (100) may receive a user's voice through a microphone (170) or an external device (e.g., a remote control). However, in some cases, the electronic device (100) may also receive a natural language command in text form through an input window displayed on the screen.

[0083] For example, when a user voice is received, the electronic device (100) can identify whether the received user voice is a preset user command. For example, if the received user command is not a preset user command, the electronic device (100) can identify the received user command as a natural language command. For example, the preset user command may be a user command in a predefined form for general control of the electronic device (100).

[0084] For example, when a user voice is received, the electronic device (100) can identify that the received user voice is a natural language command without any separate judgment action.

[0085] For convenience of explanation, the following explanation will assume that a natural language command has been received.

[0086] In operation 310, when a natural language command is received (S310:Y), in operation 320, the electronic device (100) can input the image and natural language command into an artificial intelligence model to obtain image quality adjustment information. According to one example, when a natural language command is received, the electronic device (100) can identify whether a preset keyword is included in the natural language command.

[0087] For example, if the electronic device (100) identifies that a natural language command includes the preset keyword, the electronic device (100) may input the image and natural language command into an artificial intelligence model to obtain image quality adjustment information. This is to filter whether the natural language command is an image quality-related natural language command. For example, if the natural language command includes keywords that may be related to image quality, such as clear, bright, and comfortable, the natural language command may be identified as a natural language command for image quality control. However, according to another example, the image and natural language command may be input into the artificial intelligence model without filtering whether the natural language command is an image quality-related natural language command. In this case, the artificial intelligence model may be trained to identify whether the natural language command is an image quality-related natural language command, and output image quality adjustment information if it is an image quality-related natural language command.

[0088] For example, the image quality adjustment information may include image quality adjustment items and adjustment levels. The image quality adjustment items may include at least one of brightness adjustment information, contrast adjustment information, or color adjustment information for the image.

[0089] According to one embodiment, when an image and a natural language command are input, the artificial intelligence model may be trained to acquire image quality adjustment terms (or image quality-related technical terms) based on the natural language command and output image quality adjustment information for the input image based on the image quality adjustment terms and the input image. According to one example, the artificial intelligence model may be an on-device model included in the electronic device (100), but is not limited thereto. For example, at least some functions of the artificial intelligence model may be implemented in an external device (e.g., a server).

[0090] As an example, the artificial intelligence model may include a first sub-neural network model (or first sub-network model), a second sub-neural network model (or second sub-network model), and a common neural network (or common network model) model.

[0091] The first sub-neural network model (hereinafter, "the first sub-neural network") can preprocess an input natural language command and output a preprocessed natural language command. For example, the first sub-neural network can preprocess the natural language command through at least one of tokenization, lemmatization, or stop word removal.

[0092] The second sub-neural network model (hereinafter, referred to as the second sub-neural network) can preprocess an input image and output a preprocessed image. For example, the second sub-neural network can preprocess the input image through at least one of resizing, color space conversion, or feature extraction.

[0093] A common neural network model (hereinafter, “common neural network”) can convert preprocessed natural language commands input from a first sub-neural network and preprocessed images input from a second sub-neural network into the same format, and output image quality adjustment information for the input images based on the format-converted natural language commands and the format-converted preprocessed images. For example, the common neural network can identify keyword words within the format-converted natural language commands by applying self-attention through a transformer encoder. Attention is a mechanism for focusing on important parts of a given input and processing them, and can be an operation that calculates interactions between each element of an input sequence. In other words, attention can calculate how each input element is related to other elements. For example, attention can generate a set of queries, keys, and values ​​from the input, calculate the similarity between each query and key (i.e., measure the relevance between each element of the input), and calculate a weighted sum by weighting the values ​​according to their similarity (calculating the importance of each element of the input).

[0094] As an example, a common neural network can be trained to obtain image quality adjustment information based on a format-converted image and identified keyword words.

[0095] In operation 330, the electronic device (100) can adjust the image quality of the displayed image based on the acquired image quality adjustment information.

[0096] According to one embodiment, the electronic device (100) may update image quality adjustment information based on at least one of context information of the electronic device (100) or user usage history information, and adjust the image quality of a displayed image based on the updated image quality adjustment information. The context information of the electronic device (100) may include at least one of an internal context or an external context of the electronic device (100). The user usage history information may include at least one of the user's real-time usage information, the user's past usage information, or the user's habit information.

[0097] Contextual information can include various types of information, such as device information, time information, environmental information, location information, and physiological status information. For example, device information may include information about the device used by the user, such as the device type, screen size, and operating system. For example, time information may include time-related information, such as the current time, day of the week, and season. Environmental information may include information about the surrounding environment, such as weather information, temperature information, and lighting information. For example, physiological status information may include health-related data or physiological information, such as heart rate, sleep patterns, and activity level, obtained through wearable devices.

[0098] According to one example, the electronic device (100) can obtain updated image quality adjustment information by inputting at least one of the context information of the electronic device (100) or the user's usage history information, image quality adjustment information, and image feature information into a regression model. For example, the regression model can be trained to output updated image quality adjustment information through regression analysis when at least one of the context information of the electronic device (100) or the user's usage history information, image quality adjustment information, and image feature information is input.

[0099] According to one embodiment, the electronic device (100) may train an artificial intelligence model based on user reactions received after an image whose quality has been adjusted according to image quality adjustment information is displayed. The user reactions may include various forms of user reactions that may be received after an image whose quality has been adjusted is displayed. For example, the user reactions may include at least one of a reaction without additional input after an image whose quality has been adjusted is displayed, an additional image quality adjustment reaction, a reaction indicating satisfaction with the image quality adjustment, or a reaction indicating dissatisfaction with the image quality adjustment. For example, an additional image quality adjustment reaction may include a natural language command such as "A little clearer." For example, a reaction indicating satisfaction with the image quality adjustment may include a natural language command such as "Okay, that's the quality I want." For example, a reaction indicating dissatisfaction with the image quality adjustment may include a natural language command such as "I'm not satisfied, but I'll just watch it." However, user reactions do not necessarily have to be natural language commands and may include various types of user reactions, such as button presses, receiving remote control signals, user facial expressions, user gestures, and user actions (e.g., closing curtains).

[0100] According to one embodiment, the electronic device (100) can compare images whose quality has been adjusted according to quality adjustment information, identify at least one area in the image whose quality has been adjusted, and identify information on changes in quality before and after quality adjustment for the identified at least one area. In this case, the electronic device (100) can utilize natural language commands and quality change information based on user reactions to train an artificial intelligence model.

[0101] Meanwhile, in Fig. 3, the order is mapped for all steps for convenience of explanation, but it is of course not necessarily limited to the order of steps that are not related to the order or can be performed in parallel.

[0102] FIG. 4a, FIG. 4b, FIG. 5 and FIG. 6 are drawings for explaining a learning method of an artificial intelligence model according to one or more embodiments.

[0103] According to one embodiment, the electronic device (100) can use an artificial intelligence model to understand a user's natural language commands and adjust image quality settings. Accordingly, the artificial intelligence model can be trained based on data related to the user's natural language commands and image quality settings.

[0104] According to an example, as illustrated in FIG. 4a, a data set for training an artificial intelligence model may include a training image (411), image quality adjustment information (412), and a natural language command (413) for image quality adjustment.

[0105] The image quality adjustment information (412) may include image quality element parameters of the training image (411) corresponding to the natural language command (413). For example, if a natural language command B for image quality adjustment is input while the training image A is displayed and the image quality element parameter C is adjusted accordingly, A, B, and C may be included in one training data set. For example, the image quality element parameter C may include image quality adjustment items and adjustment level information. For example, the data set required for training the artificial intelligence model may be directly collected from the electronic device (100) or the manufacturer of the electronic device (100), or an existing text-image data set may be used.

[0106] According to an example, as illustrated in FIG. 4b, a data set for training an artificial intelligence model may include a training image (411), image quality adjustment information (412), natural language commands for image quality adjustment (413), and context information (414).

[0107] Context information (414) may include at least one of the usage environment information (or context information) of the electronic device (100), the user's usage history information, or the user's habit information. For example, the desired image quality adjustment may vary based on the user's natural language command depending on the usage environment of the electronic device (100), the user's usage history, the user's habits, etc. Accordingly, the artificial intelligence model may be additionally trained based on information such as the context information (414).

[0108] According to one embodiment, the entity that acquires the data set for training the artificial intelligence model may be an electronic device (100), an external device, or a manufacturer, but for convenience of explanation, it will be described below that the electronic device (100) acquires the data set.

[0109] According to FIG. 5, the electronic device (100) can analyze a learning image to obtain feature information of the image. The feature information of the image may be in the form of a feature vector that can be used for training an artificial intelligence model. For example, the artificial intelligence model may be implemented as an LLM (Large Language Model), in which case the feature information of the image may be LLM feature information that can be input into the LLM.

[0110] For example, the electronic device (100) can capture (511, 512) high-resolution images and low-resolution images corresponding to learning images. For example, the electronic device (100) can analyze (513) the clarity / noise of the captured high-resolution image to obtain first feature information. For example, the electronic device (100) can analyze (514) the clarity / noise of the captured low-resolution image (511) to obtain second feature information and analyze (515) saliency to obtain third feature information. Saliency may be a technology for highlighting and identifying important parts that a user may be interested in within an image.

[0111] According to an example, the electronic device (100) can obtain an image with adjusted image quality by simulating (516) the image quality adjustment as illustrated in FIG. 5.

[0112] For example, the electronic device (100) can display the original and quality-adjusted versions of a learning video to obtain consumer evaluation information. For example, consumer language can be obtained based on the consumer's comparative and subjective evaluation of the differences between the original and quality-adjusted versions. For example, if a consumer evaluates the quality-adjusted version as more comfortable than the original version, the consumer language "comfortable" can be obtained.

[0113] According to one example, the electronic device (100) can train an artificial intelligence model (520) based on feature information (411) obtained from a training image, information on the amount of change in each image quality element according to image quality adjustment simulation (412), and consumer-perceived image quality (e.g., consumer language in text form) (413) for each image quality element. For example, the electronic device (100) can be trained using the feature information (411), information on the amount of change in each image quality element according to image quality adjustment simulation (412), and consumer-perceived image quality (413) for each image quality element as input data, and corresponding image quality adjustment information (521) as output data. The image quality adjustment information (521) can include image quality adjustment items and adjustment levels. For example, the image quality adjustment items can include at least one of brightness adjustment information, contrast adjustment information, and color adjustment information for the image.

[0114] However, although FIG. 5 illustrates the process of capturing images of different resolutions, such as high-resolution images and low-resolution images, to obtain different types of feature information, the present invention is not necessarily limited to this. For example, it is of course possible to capture images of specific resolutions to obtain different types of feature information.

[0115] As described above, it is possible to obtain a learning data set by collecting the language in which consumers express their feelings while watching TV screens, but this is only an example.

[0116] In another example, machine learning techniques can be used to automatically learn the relationship between language and image quality settings from previously collected data. This requires a sufficient amount of training data, and obtaining this data may require direct feedback from consumers. In another example, language and image quality settings can be mapped based on predefined rules, and user-optimized learning can be performed. For example, as illustrated in Figure 6, a data set (611 to 616) can be acquired regarding the correlation between how consumers' image quality needs are expressed in natural language, utilizing the knowledge and experience of experts (e.g., language experts and / or image quality experts).

[0117] Another example is crowdsourcing data on consumer discussions about image quality in online communities or social media. However, appropriate filtering can be performed to maintain data quality and consistency.

[0118] FIG. 7a is a diagram for explaining a method for obtaining image quality adjustment information according to one or more embodiments.

[0119] According to FIG. 7A, when a user's voice is input while an image (711) is displayed, the electronic device (100) can obtain text corresponding to the user's voice. For example, the user's voice may include a natural language command for image quality adjustment. For example, the electronic device (100) can preprocess the received audio signal (e.g., noise removal, segmentation, normalization, etc.) and convert the preprocessed audio data into text data. For example, the electronic device (100) can convert the preprocessed audio data into text data using Automatic Speech Recognition (ASR) technology.

[0120] According to one embodiment, the electronic device (100) can obtain image quality adjustment information by inputting an input image and a user's natural language command into an artificial intelligence model. For example, the artificial intelligence model may be a model that learns the relationship between image data and image quality setting values.

[0121] As an example, the artificial intelligence model may include a first sub-neural network (712), a second sub-neural network (713), and a common neural network (714), as illustrated in FIG. 7a.

[0122] For example, the electronic device (100) may preprocess a natural language command converted into text using the first sub-neural network (712). For example, the first sub-neural network (712) may preprocess an input text and convert it into specialized terminology related to image quality settings. For example, if the natural language sentence "I want a clearer image quality" is input, the first sub-neural network (712) may convert the input natural language sentence into the specialized term "increase clarity" and output it.

[0123] For example, the electronic device (100) may preprocess an input image using a second sub-neural network (713). For example, the second sub-neural network (713) may preprocess the input image data to convert its format (or form). For example, the second sub-neural network (713) may adjust the brightness or contrast of the image data to fit the input range of the common neural network (714), or, if necessary, perform preprocessing such as dimensionality reduction through a feature extraction process.

[0124] For example, the electronic device (100) may obtain image quality adjustment information for an input image based on a preprocessed natural language command and a preprocessed image using a common neural network (714). For example, the electronic device (100) may convert the preprocessed natural language command and the preprocessed image into the same format and obtain image quality adjustment information for the input image based on the format-converted natural language command and the format-converted preprocessed image. For example, the common neural network (714) may output an optimal image quality setting value based on the preprocessed text and the preprocessed image data.

[0125] For example, the image quality adjustment information output from the common neural network (714) may include image quality adjustment items and adjustment levels. The image quality adjustment items may include at least one of brightness adjustment information, contrast adjustment information, or color adjustment information for the image. For example, the image quality adjustment information may include adjustment values ​​(or control values) of multiple image quality parameters. For example, the multiple image quality parameters may be trained to minimize differences from values ​​preset in the electronic device (100). For example, when the image quality of the original image is adjusted by the adjustment value output by the common neural network (714), the original image may be trained to have a desired image quality change. For example, when the user's natural language command is "I want a clearer image quality," the common neural network (714) may output an optimal sharpness value for image data related to a corresponding image quality setting term, for example, "increase sharpness."

[0126] FIG. 7b is a diagram for explaining a method for obtaining image quality adjustment information according to one or more embodiments.

[0127] In one embodiment, the artificial intelligence model may obtain optimal image quality settings by connecting the output results of the common neural network (714) to a regression model. Accordingly, the artificial intelligence model may include a regression model (716), as illustrated in FIG. 7B . For example, the regression model may perform regression analysis using at least one regression analysis method among linear regression, decision tree, and random forest.

[0128] For example, the image quality adjustment values ​​obtained through the common neural network (714) may be predicted values ​​based on information extracted from text and image data. However, these values ​​may not yet be optimized. For example, if the user's image quality requirement is "brighter," the common neural network (714) may predict an adjustment value that increases the brightness of the image. However, simply increasing brightness may not improve the overall image quality. In some cases, complex adjustments, such as increasing contrast or adjusting color saturation, may be required to improve the overall image quality. A regression model can obtain optimized image quality settings by considering these complex factors. This is because the regression model can obtain optimal values ​​by considering various image quality parameters and their interactions. Accordingly, by optimizing the image quality settings obtained through the common neural network (714) through the regression model (716), the user's image quality requirements can be maximized while improving the overall image quality. The inputs of the regression model (716) may be text data and image data processed by the common neural network (714). Text data processed by the common neural network (714) may include specialized terminology related to image quality settings, and image data processed by the common neural network (714) may include image feature information. The output of the regression model (716) is an optimal image quality setting value for the input data, and may include adjustment values ​​necessary for the user to adjust the image quality as desired.

[0129] According to an example, the regression model (716) may be trained based on at least one of context information of the electronic device (100) or user usage history information. The context information of the electronic device (100) may include at least one of an internal context or an external context of the electronic device (100). The user usage history information may include at least one of the user's real-time usage information, the user's past usage information, or the user's habit information.

[0130] FIG. 8 is a diagram illustrating the operation of a first sub-neural network according to one or more embodiments.

[0131] According to one embodiment, the first sub-neural network (712) may preprocess input text, i.e., natural language commands. For example, the first sub-neural network (712) may preprocess the input text through at least one of tokenization, lemmatization, word embedding, text embedding, or stop word removal, as illustrated in FIG. 8 .

[0132] Tokenization can be a process in natural language processing (NLP) to break text data into analyzable units. These units are called tokens, which can typically be words, sentences, paragraphs, or other small text fragments.

[0133] Lemmatization can be the process of finding the basic form (lemma) of a word. For example, words like "running," "ran," and "runs" all have the basic form "run," and can all be converted to "run."

[0134] Word embedding can be a technique in natural language processing (NLP) that converts words into fixed-size real-valued vectors. These vectors can numerically represent the meaning of words, enabling machine learning algorithms to process text data.

[0135] Text embedding can be a technique in natural language processing (NLP) for converting text data into fixed-size real-valued vectors. Text embeddings can be generated for various text units, such as words, sentences, paragraphs, or entire documents.

[0136] Stop word removal can be the process of removing words that are not significant in the overall meaning of the text.

[0137] FIGS. 9A and 9B are diagrams illustrating the operation of a second sub-neural network according to one or more embodiments.

[0138] According to one embodiment, the second sub-neural network (713) can preprocess the input image. For example, the second sub-neural network (713) can preprocess the input image through at least one of resizing, color space conversion, or feature extraction. According to one example, the second sub-neural network (713) can be implemented as a VGG (Very Deep Convolutional Networks for Large-Scale Image Recognition) model as illustrated in FIG. 9A, but is not limited thereto. For example, the second sub-neural network (713) can be implemented as a VGG16 model or a VGG19 model. VGG16 can be composed of 13 convolutional layers and 3 fully connected layers as illustrated in FIG. 9B.

[0139] FIG. 10 is a diagram illustrating the operation of a common neural network according to one or more embodiments.

[0140] According to one embodiment, the common neural network (714) may output image quality adjustment information based on preprocessed natural language commands input from the first sub-neural network (712) and preprocessed images input from the second sub-neural network (713). For example, the common neural network (714) may be composed of various layers such as a dense layer, a convolutional layer, and a recurrent layer.

[0141] For example, the common neural network (714) may be implemented as a multi-input model capable of receiving data of different types (or formats). For example, as illustrated in FIG. 10, the multi-input model may be composed of multiple sub-networks to process different types of input data, and their outputs may be integrated to output a final result.

[0142] In general, text and images can have different formats and dimensions. For example, text is sequence data, typically vectorized using one-hot encoding or embedding techniques. In contrast, images are matrix or tensor data, which can be composed of RGB values ​​or feature maps.

[0143] In order to process different types of data in a single network, the common neural network (714) can convert the preprocessed natural language commands input from the first sub-neural network (712) and the preprocessed images input from the second sub-neural network (713) into the same format (or the same type). For this purpose, the common neural network (714) can use a convolution technique or a pooling technique. Convolution is a technique that extracts information from neighboring areas using a small kernel, and can be applied to text data as well as image data, and can be used to measure the similarity between words. Pooling is a technique that extracts the maximum or average value from the feature map resulting from the convolution, and can be applied to text data as well as image data, and can be used to compress and extract important information in a sequence. The common neural network (714) can obtain optimal image quality settings by processing text and images, that is, different types of data, into the same format.

[0144] For example, the input data of the common neural network (714) may be text data and video data. For example, the text data may be natural language sentences in which the user expresses their image quality requirements in natural language. For example, natural language commands may include natural language sentences such as "Show me a warm feeling" or "Show me bright." These natural language sentences may be raw data before being converted into specialized terminology related to image quality settings. The video data may be images of content the user is watching (e.g., TV programs, movies, etc.).

[0145] For example, the output data of the common neural network (714) may be a quality setting value. This is a value predicted by the common neural network (714) based on text data and image data, and may be used to adjust the quality setting value of the electronic device (100). For example, the quality setting value may include various quality elements such as brightness, contrast, and color.

[0146] For example, the common neural network (714) may include a Language Model (LLM), a deep learning model for natural language processing. LLM is a model that understands and generates language using large amounts of text data, and can be used to recognize a user's voice or text commands.

[0147] According to one embodiment, the common neural network (714) can identify keyword words within the format-converted natural language commands by applying self-attention via a transformer encoder. Subsequently, the common neural network (714) can obtain image quality adjustment information based on the format-converted images using the identified keyword words and a regression model.

[0148] For example, a Transformer Encoder can be used to understand the context of a natural language command, and the output of the Transformer Encoder can pass through a self-attention mechanism. Self-attention functions to emphasize important words in a natural language command, and the output of self-attention can be passed to the output layer. The output layer can utilize a multi-regression model to ultimately determine the image quality adjustment items and adjustment levels. In the loss function calculation and backpropagation process, the loss function is calculated to calculate the model error, and the model weights can be updated through backpropagation. As this process is repeated, a common neural network (714) can be trained.

[0149] For example, Attention, the output of a Transformer network, can play a crucial role in understanding the context of natural language commands. Attention emphasizes important words in natural language commands, which can help the common neural network (714) accurately understand the natural language commands. A multi-regression model can receive the output of Attention and ultimately determine the image quality adjustment items and adjustment levels. The multi-regression model is a method for modeling the relationship between multiple variables, and can learn the relationship between the output of Attention and the image quality adjustment items and adjustment levels. Accordingly, an AI model utilizing a Transformer network and a multi-regression model can adjust the image quality of an input image based on the input image and natural language commands.

[0150] For example, the results learned by the common neural network (714) may be essential semantic information extracted from text and image data. Depending on the input image and the user's image quality requirement, the common neural network (714) may extract various feature information about the image. For example, the common neural network (714) may extract information such as brightness, contrast, color distribution, and texture of the image. In addition, for the user's image quality requirement (e.g., clearer), the common neural network (714) may predict adjustment values ​​(e.g., brightness +5%, contrast +10%, etc.) to satisfy the requirement. The extracted feature information and predicted adjustment values ​​are input into a regression model, and the regression model may obtain optimal image quality settings by considering various image quality parameters and their interactions. Accordingly, the results learned by the common neural network (714) may be essential information extracted from text and image data and image quality adjustment values ​​according to the user's image quality requirement. This information may be used in the next step, the regression model, to obtain optimal image quality settings. The extracted feature information and predicted adjustment values ​​are input to a regression model (716), which can obtain optimal image quality settings by considering various image quality parameters and their interactions. Accordingly, the results learned by the common neural network (714) may be essential information extracted from text and image data and image quality adjustment values ​​according to the user's image quality requirements. This information can be used to obtain optimal image quality settings in the regression model (716).

[0151] FIG. 11 and FIG. 12 are drawings for explaining the operation of a regression model according to one or more embodiments.

[0152] In Fig. 11, for convenience of explanation, it is assumed that there are two image quality parameters, X1 and X2. As an example, a point can be determined on a plane by X1 and X2 according to the values ​​of the two image quality parameters. For example, Y can be an image quality correlation adjustment parameter of 1 and X2. Y can be defined as an image quality level according to the values ​​of each parameter X1 and X2. For example, there is a relationship that X1 should not be large or small but should maintain an intermediate value, and X2 should be larger than a certain level for the Y value to reach a peak. For example, according to the illustrated regression model, the relationship between two parameters can be extended to a relationship between more parameters so that the estimated image quality parameters can be adjusted according to the target image quality level (image quality rule).

[0153] According to one embodiment, the regression model may be implemented in a form including an input layer, a hidden layer, and an output layer, as illustrated in FIG. 12. For example, since the regression model is a multiple image quality parameter adjustment model, it may be implemented as a multi-regression model.

[0154] For example, a regression model can be trained to adjust input parameters through error backpropagation. Error backpropagation can work by comparing the output of a neural network for a given input with the actual target output, and propagating the resulting error through each layer of the network. For example, a regression model can be trained to estimate a Y value through error backpropagation, and the difference between the estimated Y value and a preset Y' value (Y-Y') becomes an error, and the input parameters are adjusted until the error is minimized. For example, the preset Y' value can be a quality level value determined by the system or set by the designer.

[0155] FIG. 13, FIG. 14 and FIG. 15 are drawings for explaining a method for obtaining image quality adjustment information according to one or more embodiments.

[0156] In one embodiment, the artificial intelligence model may further include a feature evaluation model (717), as illustrated in FIG. 13. In one example, the feature evaluation model may compare image features of an input image and an image whose quality has been adjusted to a threshold value or higher, thereby identifying an image region in which the quality has been adjusted to a threshold value or higher.

[0157] For example, the electronic device (100) may interpret a user's natural language command using a feature evaluation model (717) as illustrated in FIG. 14, and detect the level of feature change in each image region based on the effect of the parameter that adjusts the image quality accordingly. The process of comparing images before and after a natural language image quality command to detect the level of change in important features of the image may be similar to the process of conventional image analysis. For example, features such as contrast ratio, sharpness, colorfulness, skin tone, and noise level of the image before and after image quality adjustment may be analyzed for each region.

[0158] For example, as illustrated in FIG. 15, the above-described operations may be repeatedly performed as user commands are repeatedly received. For example, the electronic device (100) may repeatedly identify the user's natural language image quality command and the level of image quality change resulting from the corresponding parameter adjustments using the feature evaluation model (717) until the user is satisfied (e.g., until image quality adjustment is completed). For example, the information "user command vs. image quality feature change level" may indicate a correlation between the exact level of image quality adjustment required by the user's natural language command.

[0159] According to one embodiment, the electronic device (100) may utilize transfer learning to identify a user's desired image quality adjustment request with minimal commands for training a feature evaluation model (717). Transfer learning may be a machine learning technique that utilizes a previously trained model to apply new features or upgrades. For example, information indicating the correlation between a user's natural language command and the corresponding image quality adjustment level may be utilized in transfer learning.

[0160] For example, when transfer learning is used to train a feature evaluation model (717), initial weights can be set using a model previously trained on a large dataset, and fine-tuning can be performed based on new data. For example, weight fine-tuning can be performed based on the relationship between a user's natural language commands and the image quality adjustment level. Using transfer learning, knowledge from previously trained models can be leveraged to create a more accurate and efficient personalized image quality adjustment model.

[0161] Although the various embodiments described above have been described as utilizing multiple individual neural network models, the operations of at least two of the multiple neural network models may be implemented in a single neural network model.

[0162] Each operation according to the various embodiments described above may be performed by the processor (140), but if necessary, a module for each operation may be utilized. For example, each module may be implemented using at least one software, at least one hardware, and / or a combination thereof. Each module may be implemented to utilize a predefined algorithm, a predefined formula, and / or a learned artificial intelligence model to perform the operation. However, at least some modules may be distributed to an external device.

[0163] According to the various embodiments described above, the existing complex picture quality setting process can be replaced by allowing users to simply adjust the picture quality using natural language. Furthermore, by enabling detailed picture quality settings tailored to the individual user's preferences and viewing environment, the existing limited picture quality adjustment UX and limited range of choices can be overcome. Furthermore, by providing quick and easy operability, the existing cumbersome and time-consuming operation process is simplified and the user's convenience in picture quality adjustment is increased, thereby enhancing the TV viewing experience. Furthermore, by responding to spontaneous picture quality adjustment requests, the system can provide high-quality video content that meets user standards, thereby improving consumer satisfaction.

[0164] Meanwhile, the methods according to the various embodiments of the present disclosure described above can be implemented only with a software upgrade or a hardware upgrade for an existing electronic device and / or server.

[0165] Additionally, the various embodiments of the present disclosure described above can also be performed through an embedded server provided in an electronic device, or an external server of the electronic device.

[0166] Meanwhile, according to a temporary example of the present disclosure, the various embodiments described above can be implemented as software including instructions stored in a machine-readable storage medium that can be read by a machine (e.g., a computer). The device is a device that can call instructions stored from the storage medium and operate according to the called instructions, and may include an electronic device (e.g., electronic device (A)) according to the disclosed embodiments. When an instruction is executed by a processor, the processor can perform a function corresponding to the instruction directly or by using other components under the control of the processor. The instruction may include code generated or executed by a compiler or interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' means that the storage medium does not contain a signal and is tangible, but does not distinguish between data being stored semi-permanently or temporarily in the storage medium.

[0167] Furthermore, according to one embodiment of the present disclosure, the method according to the various embodiments described above may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or online through an application store (e.g., Play Store™). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0168] In addition, each of the components (e.g., modules or programs) according to the various embodiments described above may be composed of a single or multiple entities, and some of the corresponding sub-components described above may be omitted, or other sub-components may be further included in various embodiments. Alternatively or additionally, some components (e.g., modules or programs) may be integrated into a single entity, which may perform the same or similar functions as those performed by each of the corresponding components prior to integration. Operations performed by modules, programs or other components according to various embodiments may be executed sequentially, in parallel, iteratively or heuristically, or at least some operations may be executed in a different order, omitted, or other operations may be added.

[0169] Although the preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above, and various modifications may be made by those skilled in the art to which the present disclosure pertains without departing from the gist of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical idea or prospect of the present disclosure. It will also be understood that any embodiment(s) described in the present disclosure may be used in conjunction with any other embodiment(s) described in the present disclosure.

Claims

1. In electronic devices, display; memory that stores instructions; and one or more processors including processing circuitry; One or more of the above processors, When the above instructions are executed individually or collectively, When a voice input is received while an image is displayed through the above display, the image and the voice input are input into an artificial intelligence model to obtain image quality adjustment information, Adjusting the image quality of the image displayed through the display based on the acquired image quality adjustment information; The above artificial intelligence model, An electronic device, which is trained to output image quality adjustment information for the input image based on the image and voice input and image quality adjustment terms corresponding to the voice input.

2. In paragraph 1, The one or more processors, when the instructions are executed individually or collectively, An electronic device that updates the image quality adjustment information based on at least one of context information of the electronic device or user usage history information, and adjusts the image quality of an image displayed through the display based on the updated image quality adjustment information.

3. In paragraph 2, The one or more processors, when the instructions are executed individually or collectively, At least one of the context information of the electronic device or the user's usage history information, the image quality adjustment information, and the image feature information is input into a regression model to obtain the updated image quality adjustment information. The above regression model is, An electronic device that is trained to output the updated image quality adjustment information through regression analysis when at least one of the context information of the electronic device or the user's usage history information, the image quality adjustment information, and the image characteristic information is input.

4. In paragraph 1, The above artificial intelligence model, An electronic device that learns based on user reactions received after an image whose quality has been adjusted according to the above quality adjustment information is displayed through the display.

5. In paragraph 4, The one or more processors, when the instructions are executed individually or collectively, Comparing the image and the image whose quality has been adjusted according to the image quality adjustment information to identify at least one area in the image whose quality has been adjusted, Identifying image quality change information before and after image quality adjustment for at least one area identified above, An electronic device that uses the voice input and the image quality change information according to the user reaction to learn the artificial intelligence model.

6. In paragraph 1, The above artificial intelligence model, A first sub-neural network model that preprocesses the input voice input and outputs the preprocessed voice; A second sub-neural network model that preprocesses the input image and outputs a preprocessed image; and An electronic device comprising a common neural network model that converts the preprocessed voice input from the first sub-neural network model and the preprocessed image input from the second sub-neural network model into the same format and outputs image quality adjustment information for the input image based on the format-converted voice and the format-converted preprocessed image.

7. In paragraph 6, The above artificial intelligence model, Self-Attention is applied through a Transformer Encoder to identify keyword words within the converted speech, An electronic device that learns to obtain the image quality adjustment information based on the format-converted image and the identified keyword words using a regression model.

8. In paragraph 6, The above first sub-neural network model is, Preprocessing the speech input through at least one of tokenization, lemmatization, or stop word removal for the speech input, The above second sub-neural network model is, An electronic device that preprocesses the input image through at least one of resizing, color space conversion, or feature extraction.

9. In paragraph 1, The one or more processors, when the instructions are executed individually or collectively, When the voice input is received while the image is displayed through the display, it is identified whether the voice input includes a preset keyword, An electronic device that, when the voice input is identified as including the preset keyword, inputs the image and the voice input into the artificial intelligence model to obtain image quality adjustment information.

10. In paragraph 1, The above image quality adjustment information is: Includes image quality adjustment items and adjustment levels, The above image quality adjustment items are: An electronic device comprising at least one of brightness adjustment information, contrast adjustment information, or color adjustment information for the image.

11. In a method for controlling an electronic device, When a voice input is received while an image is displayed, a step of inputting the image and the voice input into an artificial intelligence model to obtain image quality adjustment information; and A step of adjusting the image quality of the displayed image based on the acquired image quality adjustment information; The above artificial intelligence model, A control method, which is learned to output image quality adjustment information for the input image based on the image and voice input and image quality adjustment terms corresponding to the voice input.

12. In paragraph 11, The steps for adjusting the image quality of the image displayed above are: A step of updating the image quality adjustment information based on at least one of context information of the electronic device or usage history information of the user; and A control method comprising: a step of adjusting the image quality of the displayed image based on the updated image quality adjustment information.

13. In paragraph 12, The steps for updating the above picture quality adjustment information are: At least one of the context information of the electronic device or the user's usage history information, the image quality adjustment information, and the image feature information is input into a regression model to obtain the updated image quality adjustment information. The above regression model is, A control method that is learned to output the updated image quality adjustment information through regression analysis when at least one of the context information of the electronic device or the user's usage history information, the image quality adjustment information, and the image characteristic information is input.

14. In paragraph 11, The above artificial intelligence model, A control method in which a video whose quality has been adjusted according to the above quality adjustment information is displayed through the display and then learned based on user reactions received.

15. A non-transitory computer-readable medium storing computer instructions that, when executed by a processor of an electronic device, cause the electronic device to perform an operation, The above action is, When a voice input is received while an image is displayed, a step of inputting the image and the voice input into an artificial intelligence model to obtain image quality adjustment information; and A step of adjusting the image quality of the displayed image based on the acquired image quality adjustment information; The above artificial intelligence model, A non-transitory computer-readable medium, which is trained to output image quality adjustment information for the input image based on the image and voice input and image quality adjustment terms corresponding to the voice input.

Images