Electronic device and operating method of electronic device

Abstract

This electronic device comprises: an illuminance sensor; a camera; a memory storing instructions; and a processor, wherein the processor executes the instructions to control the electronic device to: on the basis of segmentation information based on at least one object and a background, which are included in an image, and gaze information acquired through the camera, determine at least one element of interest among the at least one object and the background; acquire concentration information corresponding to the at least one element of interest through a first artificial intelligence model, on the basis of at least one face image acquired through the camera, brightness information of the image, and illuminance information acquired through the illuminance sensor; and correct the luminance of the image according to a correction weight acquired through a second artificial intelligence model, on the basis of the element of interest and the concentration information.

Description

Electronic devices and methods of operating electronic devices

[0001] The present disclosure relates to an electronic device and an operating method thereof. Specifically, the present disclosure relates to an electronic device for obtaining a corrected image in which the luminance of the image is corrected, and a method of operating the electronic device.

[0002] An artificial intelligence system is a computer system that implements human-level intelligence. It is a system in which the machine learns and makes judgments on its own, and its recognition rate improves with use.

[0003] It consists of element technologies that mimic the cognitive and judgment functions of the human brain by utilizing machine learning technology and machine learning algorithms.

[0004] The element technologies may include, for example, at least one of a linguistic understanding technology that recognizes human language / characters, an inference / prediction technology that judges information and logically infers and predicts, and a knowledge representation technology that processes human experience information into knowledge data.

[0005] Additionally, recent advancements in technology related to electronic devices that display images and provide them to users have been observed. For example, the brightness of specific areas of images displayed on electronic devices can be enhanced and provided to users.

[0006] For example, the luminance of the central region of an image can be emphasized and presented to the user. Additionally, the luminance of objects included in the image can be emphasized and presented to the user.

[0007] One embodiment of the present disclosure provides an electronic device. The electronic device may include an illumination sensor. The electronic device may include a camera. The electronic device may include a memory storing at least one instruction. The electronic device may include at least one processor that executes at least one instruction stored in the memory. By executing at least one instruction by the at least one processor, the electronic device may determine at least one element of interest among at least one object and at least one background based on segmentation information based on at least one object and background included in an image and gaze information acquired through a camera. By executing at least one instruction by the at least one processor, the electronic device may obtain concentration information corresponding to at least one element of interest through a first artificial intelligence model based on at least one face image acquired through the camera, brightness information of the image, and concentration information acquired through an illumination sensor. By executing at least one instruction by the at least one processor, the electronic device may control the luminance of the image to be corrected according to a correction weight acquired through a second artificial intelligence model based on the element of interest and concentration information.

[0008] As one embodiment of the present disclosure, a method of operating an electronic device may be provided. The method of operating an electronic device may include a step of determining an element of interest of a user among at least one object and a background based on segmentation information based on at least one object and a background included in an image and gaze information acquired through a camera. The method of operating an electronic device may include a step of acquiring concentration information corresponding to at least one element of interest through a first artificial intelligence model based on at least one user's face image acquired through a camera, brightness information of the image, and illuminance information acquired through an illuminance sensor. The method of operating an electronic device may include a step of acquiring a corrected image in which the brightness of the image is corrected according to a correction weight acquired through a second artificial intelligence model, based on the element of interest and the concentration information.

[0009] As one embodiment of the present disclosure, a computer-readable recording medium having recorded thereon a program for performing at least one method of the method of operating the disclosed electronic device on a computer may be provided.

[0010] The technical problems to be achieved in this document are not limited to the technical problems mentioned above, and other technical problems not mentioned can be clearly understood by a person having ordinary skill in the technical field to which the present disclosure belongs from the description below.

[0011] The present disclosure may be understood in conjunction with the following detailed description and accompanying drawings, wherein reference numerals refer to structural elements.

[0012] FIG. 1 is a drawing for explaining the operation of an electronic device according to one embodiment of the present disclosure.

[0013] FIG. 2 is a block diagram illustrating the configuration of an electronic device according to one embodiment of the present disclosure.

[0014] FIG. 3 is a flowchart illustrating the operation of an electronic device according to one embodiment of the present disclosure.

[0015] FIG. 4 is a drawing for explaining the operation of an electronic device according to one embodiment of the present disclosure.

[0016] FIG. 5 is a diagram for explaining an operation of obtaining segmentation information by segmenting an object and a background included in an image according to one embodiment of the present disclosure.

[0017] FIG. 6 is a diagram for explaining an operation of acquiring a user's face image and user's gaze information according to one embodiment of the present disclosure.

[0018] FIG. 7 is a diagram for explaining an operation of obtaining illumination information of an electronic device according to one embodiment of the present disclosure.

[0019] FIG. 8 is a diagram for explaining an operation of obtaining concentration information on a user's interest element through a first artificial intelligence model according to one embodiment of the present disclosure.

[0020] FIG. 9 is a flowchart illustrating an operation of obtaining a transformed face image according to one embodiment of the present disclosure.

[0021] FIG. 10 is a diagram for explaining an operation of obtaining object weights and background weights through a second artificial intelligence model according to one embodiment of the present disclosure.

[0022] FIG. 11 is a flowchart illustrating an operation of obtaining image weights through a second artificial intelligence model and obtaining a corrected image based on object weights, background weights, and image weights, according to one embodiment of the present disclosure.

[0023] FIG. 12 is a diagram for explaining a plurality of nodes and a plurality of edges included in graph information according to one embodiment of the present disclosure.

[0024] FIG. 13 is a diagram for explaining an operation of training a second artificial intelligence model according to one embodiment of the present disclosure.

[0025] FIG. 14 is a flowchart for explaining an operation of displaying a correction image through a display according to one embodiment of the present disclosure.

[0026] The terms used in this disclosure will be briefly explained, and one embodiment of the present disclosure will be specifically described.

[0027] Throughout this disclosure, unless specifically stated otherwise, "or" is inclusive and not exclusive. Thus, unless explicitly stated otherwise or context dictates otherwise, "A or B" may refer to "A, B, or both."

[0028] In this disclosure, the expression “at least one of a, b or c” may refer to “a”, “b”, “c”, “a and b”, “a and c”, “b and c”, “all of a, b and c”, or variations thereof.

[0029] The terms used in this disclosure are selected from widely used, current terms, taking into account the functions of one embodiment of the disclosure. However, these terms may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, terms may be arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the description of the relevant embodiments of the disclosure. Therefore, the terms used in this disclosure should not be defined simply as names of terms, but rather based on the meanings of the terms and the overall content of the disclosure.

[0030] Singular expressions may include plural expressions unless the context clearly indicates otherwise. Terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art described herein.

[0031] Throughout this disclosure, when a part is said to "include" a component, this does not exclude other components, but rather implies the inclusion of other components, unless otherwise specifically stated. Furthermore, terms such as "part," "module," and the like described herein refer to a unit that processes at least one function or operation, which may be implemented in hardware or software, or a combination of hardware and software.

[0032] The expression “configured to” as used herein can be used interchangeably with, for example, “suitable for,” “having the capacity to,” “designed to,” “adapted to,” “made to,” or “capable of.” The term “configured to” does not necessarily mean something that is “specifically designed to” in terms of hardware. Instead, in some contexts, the expression “a system configured to” can mean that the system is “capable of” doing something together with other devices or components. For example, the phrase “a processor configured (or set) to perform A, B, and C” may mean a dedicated processor (e.g., an embedded processor) for performing those operations, or a generic-purpose processor (e.g., a CPU or application processor) that can perform those operations by executing one or more software programs stored in memory.

[0033] Additionally, when a component is referred to as being “connected” or “connected” to another component in the present disclosure, it should be understood that the component may be directly connected or connected to the other component, but may also be connected or connected via another component in between, unless otherwise specifically stated.

[0034] It should be understood that the blocks and combinations of flowcharts in each flowchart can be executed by one or more computer programs containing computer-executable instructions. The one or more computer programs may be stored entirely in a single memory, or may be stored in separate portions across multiple different memories.

[0035] Any function or operation described in this document may be performed by a single processor or a combination of multiple processors.

[0036] The artificial intelligence-related functions according to the present disclosure are operated via a processor and memory. One or more processors control the processing of input data according to predefined operating rules or artificial intelligence models stored in the memory. Alternatively, if one or more processors are dedicated artificial intelligence processors, the dedicated artificial intelligence processors may be designed with a hardware structure specialized for processing a specific artificial intelligence model.

[0037] The predefined operation rules or artificial intelligence model are characterized by being created through learning. Here, being created through learning means that the basic artificial intelligence model is learned by a learning algorithm using a plurality of learning data, thereby creating a predefined operation rules or artificial intelligence model set to perform a desired characteristic (or purpose). This learning may be performed in the electronic device itself using the artificial intelligence model according to the present disclosure, or may be performed through a separate server and / or system. Examples of the learning algorithm include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0038] An artificial intelligence model may be composed of multiple neural network layers. Each of the multiple neural network layers has multiple weight values, and performs neural network operations through operations between the operation results of the previous layer and the multiple weights. The multiple weights of the multiple neural network layers may be optimized based on the learning results of the artificial intelligence model. For example, the multiple weights may be updated so that the loss value or cost value obtained from the artificial intelligence model is reduced or minimized during the learning process. The artificial neural network may include a deep neural network (DNN), and examples thereof include, but are not limited to, a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), or deep Q-networks.

[0039] Below, with reference to the attached drawings, embodiments of the present disclosure are described in detail so that those skilled in the art can easily implement the present disclosure. However, one embodiment of the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. In addition, in the drawings, parts irrelevant to the description are omitted to clearly describe one embodiment of the present disclosure, and similar parts are designated with similar drawing reference numerals throughout the present disclosure.

[0040] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings.

[0041] FIG. 1 is a drawing for explaining the operation of an electronic device according to one embodiment of the present disclosure.

[0042] Referring to FIG. 1, in one embodiment of the present disclosure, an electronic device (100) may be a device that displays an image (200) and provides it to a user (300). The electronic device (100) may include a display that displays the image (200). The electronic device (100) may be a display device that includes a display that displays the image (200).

[0043] In one embodiment of the present disclosure, FIG. 1 illustrates an electronic device (100) in the shape of a television. However, the present disclosure is not limited thereto. The electronic device (100) may be implemented as electronic devices of various shapes, such as a mobile device, a smart phone, a laptop computer, a desktop, a tablet PC, a digital signage, a projector, and a wearable device.

[0044] In one embodiment of the present disclosure, an electronic device (100) implemented in various shapes may display an image (200) through a display to provide the image (200) to a user (300).

[0045] However, the present disclosure is not limited thereto, and the electronic device (100) may be implemented as an electronic device of various shapes, such as a desktop, a set-top box, or a server device. In this case, the electronic device (100) may receive an image (200) from an external electronic device including a display connected through an input / output interface, and provide a corrected image (230) with corrected brightness to the external electronic device or an external server. In addition, the electronic device (100) may receive an image (200) from an external server through a communication interface, and provide a corrected image (230) with corrected brightness to the external server.

[0046] In one embodiment of the present disclosure, an image (200) may include at least one object (210) and a background (220). An "object" may refer to an object, person, obstacle, etc. included in the image (200). In one embodiment of the present disclosure, an "object" may refer to a person, a car, a motorcycle, a tree, a cloud, the sun, etc., and is not limited to any one of them. An image (200) may include one or more objects.

[0047] "Background" may refer to a space in which an object (210) is located within an image (200). In one embodiment of the present disclosure, "background" may refer to a space such as the sky, the sea, the ground, a ceiling, a wall, a parking lot, etc., and is not limited thereto. The image (200) may include one or more backgrounds.

[0048] In one embodiment of the present disclosure, at least one user (300) can view an image (200) via an electronic device (100). While FIG. 1 illustrates three users viewing an image (200) via an electronic device (100), the present disclosure is not limited thereto. It goes without saying that two or fewer users, or four or more users, can view an image (200) via an electronic device (100).

[0049] In one embodiment of the present disclosure, at least one user (300) may view the image (200) by focusing his / her gaze on a specific area corresponding to a specific object or background included in the image (200). In one embodiment of the present disclosure, when two or more users view the image (200), the specific areas within the image (200) that each of the multiple users gazes at may be different.

[0050] In one embodiment of the present disclosure, the electronic device (100) may display a corrected image (230) that corrects the brightness of the image (200) through a display to increase the concentration of at least one user (300) viewing the image (200). When there are multiple users viewing the image (200), the electronic device (100) may display a corrected image (230) that corrects the brightness of the image (200) through a display so that the sum of the concentrations of the multiple users is maximized.

[0051] In one embodiment of the present disclosure, the electronic device (100) may determine, based on gaze information of at least one user (300), an object at which the gaze of at least one user (300) is located among at least one object (210) and a background (220) included in an image (200), as an “element of interest.” When multiple users view the image (200), the element of interest may include two or more objects.

[0052] In one embodiment of the present disclosure, the concentration of at least one user (300) viewing an image (200) may be obtained through a pre-trained artificial intelligence model capable of inferring the concentration of the user. In one embodiment of the present disclosure, the artificial intelligence model used to infer the concentration may be an artificial intelligence model capable of inferring the concentration of at least one user (300) viewing an image (200) based on a facial image of at least one user (300), brightness information of the image (200), and lighting information of the electronic device (100).

[0053] In one embodiment of the present disclosure, the electronic device (100) may obtain correction weights corresponding to brightness correction of an image using information on elements of interest and concentration. The correction weights may include an object weight for correcting the brightness of at least one object (210) included in the image (200) and a background weight for correcting the brightness of the background (220). In this case, the electronic device (100) may obtain the object weights and background weights using a pre-learned artificial intelligence model to infer object weights and background weights that can increase the concentration of a user (300) based on the information on elements of interest and concentration.

[0054] In one embodiment of the present disclosure, the object weight and background weight inferred through the artificial intelligence model may be weights optimized to increase the concentration of at least one user (300) when correcting the brightness of the image (300).

[0055] In one embodiment of the present disclosure, the electronic device (100) can obtain a corrected image (230) in which the brightness of the image (200) is corrected based on the obtained correction weights. The electronic device (100) can obtain a corrected image (230) in which the brightness of the image (200) is corrected based on the obtained object weights and background weights. The corrected image (230) can include an object (240) in which the brightness is corrected based on the object weights and a background (250) in which the brightness is corrected based on the background weights. The electronic device (100) can display the obtained corrected image (230) through a display and provide it to at least one user (300).

[0056] At this time, if there are two or more objects included in the image (200), the object weights may include multiple sub-object weights that correspond to different objects and have different values. In addition, if there are two or more backgrounds included in the image (200), the background weights may include multiple sub-background weights that correspond to different backgrounds and have different values. The electronic device (100) may correct the brightness of the image (200) based on the multiple sub-object weights and the multiple sub-background weights.

[0057] In one embodiment of the present disclosure, when correcting the luminance of an image (200), the electronic device (100) may make the degree of luminance correction of an image element corresponding to a large weight different from the degree of luminance correction of an image element corresponding to a small weight. In this case, the “degree of luminance correction” may mean the ratio of the luminance of the image before correction and the luminance of the image after correction. The electronic device (100) may make the degree of luminance correction of at least one object (210) and background (220) included in the image (200) different according to the acquired object weight and background weight.

[0058] In one embodiment of the present disclosure, as multiple sub-object weights are acquired, the electronic device (100) can compensate for the brightness of the image (200) by varying the degree of brightness correction for each of the multiple objects. As multiple sub-background weights are acquired, the electronic device (100) can compensate for the brightness of the image (200) by varying the degree of brightness correction for each of the multiple backgrounds.

[0059] In one embodiment of the present disclosure, the electronic device (100) provides a corrected image (230) with brightness correction to at least one user (300) based on weights obtained through a pre-learned artificial intelligence model to infer weights that can increase the user's concentration, so that the electronic device (100) can provide an image (230) having optimal brightness that can increase the concentration to at least one user (300).

[0060] At this time, when determining the concentration of at least one user (300), by using the brightness information of the image (200) and the illuminance information of the electronic device (100) as input data, the electronic device (100) can provide a corrected image (230) with the brightness corrected so as to increase the concentration of at least one user (300) by taking into consideration the brightness information of the image (200) and the illuminance information of the electronic device (100).

[0061] In addition, even when multiple users use the electronic device (100), the electronic device (100) can improve the concentration of the multiple users by providing the multiple users with a corrected image (230) whose brightness is corrected based on a weight determined using the concentration of each of the multiple users and an interest factor determined using the gaze information of the multiple users.

[0062] Hereinafter, the electronic device (100) and the operation of the electronic device (100) will be described with reference to FIGS. 2 to 14.

[0063] FIG. 2 is a block diagram illustrating the configuration of an electronic device according to one embodiment of the present disclosure.

[0064] Referring to FIGS. 1 and 2, in one embodiment of the present disclosure, an electronic device (100) may include a display (110), a memory (120), at least one processor (130), a camera (140), a light sensor (150), an input / output interface (160), and a communication interface (170).

[0065] However, not all of the components illustrated in FIG. 2 are essential components. The electronic device (100) may be implemented with more components than those illustrated in FIG. 2, or may be implemented with fewer components.

[0066] The display (110), memory (120), at least one processor (130), camera (140), light sensor (150), input / output interface (160), and communication interface (170) included in the electronic device (100) may each be electrically connected to each other.

[0067] In one embodiment of the present disclosure, the display (110) may include any one of a liquid crystal display, a plasma display, an organic light emitting diode display, and an inorganic light emitting diode display. However, the present disclosure is not limited thereto, and the display (110) may include other types of displays capable of displaying an image (200).

[0068] In one embodiment of the present disclosure, at least one processor (130) can display an image (200) or a luminance-corrected corrected image (230) through a display (110) and provide it to a user (300).

[0069] In one embodiment of the present disclosure, the memory (120) may store instructions, data structures, and program codes that can be read by at least one processor (130). In one embodiment of the present disclosure, there may be more than one memory (120). Operations performed by the electronic device (100) may be implemented by at least one processor (130) executing instructions or codes of a program stored in the memory (120).

[0070] In one embodiment of the present disclosure, the memory (120) may include at least one of a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), a RAM (Random Access Memory), a SRAM (Static Random Access Memory), a ROM (Read-Only Memory), an EEPROM (Electrically Erasable Programmable Read-Only Memory), a PROM (Programmable Read-Only Memory), a Mask ROM, a Flash ROM, etc.), a hard disk drive (HDD), or a solid state drive (SSD).

[0071] In one embodiment of the present disclosure, the memory (120) may not exist separately and may be configured to be included in at least one processor (130).

[0072] In one embodiment of the present disclosure, memory (120) may store instructions or program codes for performing functions or operations of the electronic device (100). Instructions, algorithms, data structures, program codes, and application programs stored in memory (120) may be implemented in a programming or scripting language such as, for example, C, C++, Java, Python, or an assembler.

[0073] In one embodiment of the present disclosure, various types of modules that can be used to perform operations of the electronic device (100) may be stored in the memory (120).

[0074] In one embodiment of the present disclosure, the memory (120) may store an image segmentation module (121), a face recognition module (122), an eye tracking module (123), an interest element determination module (124), a brightness information acquisition module (125), an illuminance acquisition module (126), a first artificial intelligence model (127), a second artificial intelligence model (128), and an image correction module (129). However, not all of the modules or models illustrated in FIG. 2 are essential. The memory (120) may store more modules or models than the modules or models illustrated in FIG. 2, or may store fewer modules or models.

[0075] In one embodiment of the present disclosure, a 'module' included in the memory (120) may mean a unit that processes a function or operation performed by at least one processor (130). The 'module' included in the memory (120) may be implemented as software such as instructions, an algorithm, a data structure, or a program code.

[0076] In one embodiment of the present disclosure, the 'model' included in the memory (120) may mean software such as instructions, algorithms, data structures, or program codes used to perform a specific function or operation through at least one processor (130).

[0077] In one embodiment of the present disclosure, the image segmentation module (121) may be configured with commands or program codes related to an operation or function of segmenting at least one object (210) and a background (220) included in an image (200) obtained through an input / output interface (160) or a communication interface (170) to obtain segmentation information. However, the present disclosure is not limited thereto, and the image segmentation module (121) may be configured with commands or program codes related to an operation or function of obtaining segmentation information from an image stored in a memory (120).

[0078] In one embodiment of the present disclosure, the image segmentation module (121) may include commands or program codes for performing a panoptic segmentation operation. However, the present disclosure is not limited thereto, and the image segmentation module (121) may include a pre-trained artificial intelligence model to infer segmentation information in which at least one object (210) and a background (220) included in an image (200) are segmented. At this time, the artificial intelligence model included in the image segmentation module (121) is a DNN (Deep Neural Network), and may include a CNN (Convolutional Neural Network), a U-net, an RNN (Recurrent Neural Network), a Transformer model, etc., and the artificial intelligence model in the present disclosure is not limited to the above-described examples.

[0079] At least one processor (130) can obtain segmentation information that segments at least one object (210) and background (220) included in an image (200) by executing commands or program codes of an image segmentation module (121).

[0080] When there are two or more objects included in the image (200), or when the background of the image (200) is divided into two or more, at least one processor (130) can obtain segmentation information in which each of the plurality of objects or the plurality of backgrounds included in the image (200) is divided by executing the commands or program code of the image segmentation module (121).

[0081] In one embodiment of the present disclosure, the face recognition module (122) may include commands or program codes for performing an operation of recognizing the face of at least one user (300) from an image of at least one user (300) acquired through a camera (140) and acquiring a face image.

[0082] The face recognition module (122) may include commands or program codes for performing an operation of detecting a face of at least one user (300) from an image of at least one user (300) and segmenting the detected face to obtain a face image.

[0083] At least one processor (130) can obtain a facial image of at least one user (300) from an image of at least one user (300) by executing commands or program codes of a facial recognition module (122).

[0084] In one embodiment of the present disclosure, when there are two or more users viewing the video (200), at least one processor (130) can obtain facial images of each of the plurality of users from the images of the plurality of users by executing commands or program codes of the facial recognition module (122).

[0085] In one embodiment of the present disclosure, the gaze tracking module (123) may include commands or program codes for performing an operation of obtaining gaze information of at least one user (300) based on a facial image of at least one user (300).

[0086] The gaze tracking module (123) may include commands or program codes for performing an operation of detecting a facial landmark included in a facial image, such as a jawline, eyebrows, nose, mouth outline, etc., and finding the position of the eyes of at least one user (300) using the detected facial landmarks. The gaze tracking module (123) may include commands or program codes for detecting the position of the pupil in the eyes of at least one user (300) and tracking the gaze of at least one user (300) using the position of the pupil and facial landmarks, etc., to obtain gaze information.

[0087] However, the present disclosure is not limited thereto, and the gaze tracking module (123) may of course include other methods of commands or program codes for obtaining gaze information of at least one user (300) from a facial image of at least one user (300).

[0088] In one embodiment of the present disclosure, when there are two or more users viewing the video (200), at least one processor (130) can obtain gaze information of each of the plurality of users from the facial images of each of the plurality of users by executing commands or program codes of the gaze tracking module (123).

[0089] In one embodiment of the present disclosure, the interest element determination module (124) may include instructions or program codes for determining an interest element of at least one user (300) among at least one object (210) and at least one background (220) based on segmentation information and gaze information.

[0090] The interest element determination module (124) may include commands or program codes for determining an image element where the gaze of at least one user (300) is located among at least one segmented object (210) and background (220) as an interest element based on segmentation information and gaze information.

[0091] At least one processor (130) can determine an element of interest of at least one user (300) among at least one object (210) and background (220) based on segmentation information and gaze information by executing instructions or program codes of an element of interest determination module (124). When the gaze information includes gaze information of multiple users, at least one processor (130) can determine an element of interest of each of the multiple users. In this case, the element of interest of each of the multiple users can be a different object (e.g., a specific object or a specific background) within the image (200).

[0092] In one embodiment of the present disclosure, the image segmentation module (121), the face recognition module (122), the gaze tracking module (123), and the interest element determination module (124) are illustrated as distinct modules; however, the present disclosure is not limited thereto. Depending on the operating method of the electronic device (100), a single module may of course include instructions or program codes for performing two or more operations.

[0093] In one embodiment of the present disclosure, the brightness information acquisition module (125) may include commands or program codes for performing an operation of acquiring brightness information of an image (200) based on the acquired image (200). In one embodiment of the present disclosure, the brightness information acquisition module (125) may include commands or program codes for acquiring brightness information of an image (200) based on R, G, B data or gray data of the acquired image (200).

[0094] At least one processor (130) can acquire brightness information of an image (200) based on the acquired image (200) by executing commands or program codes of a brightness information acquisition module (125).

[0095] In one embodiment of the present disclosure, the illuminance acquisition module (126) may include commands or program codes for performing an operation of acquiring illuminance information of the electronic device (100) through the illuminance sensor (150). In one embodiment of the present disclosure, the illuminance acquisition module (126) may include commands or program codes for controlling the operation of the illuminance sensor (150).

[0096] At least one processor (130) can acquire illumination information of the electronic device (100) through the illumination sensor (150) by executing commands or program codes of the illumination acquisition module (126).

[0097] In one embodiment of the present disclosure, the first artificial intelligence model (127) may be an artificial intelligence model trained to infer concentration information on an element of interest of at least one user (300) based on a facial image, brightness information, and illuminance information of at least one user (300).

[0098] In one embodiment of the present disclosure, "concentration information" may be a value that numerically represents the concentration of at least one user (300) on an element of interest. The concentration information may be a value that is numerically represented by comparing the concentration of at least one user (300) on an element of interest with the reference concentration, using the highest concentration measured among the repeatedly measured concentrations of at least one user (300) watching the video (200) as a reference concentration. In this case, the concentration may be measured by brain wave information, etc. analyzed through electrodes, etc., that measure brain waves of at least one user (300).

[0099] At least one processor (130) can provide a facial image, brightness information, and illuminance information as input data to the first artificial intelligence model (127), thereby obtaining concentration information on an element of interest of at least one user (300) as output data.

[0100] In one embodiment of the present disclosure, by providing facial images of multiple users to the first artificial intelligence model (127), at least one processor (130) may obtain concentration information on each of the multiple users' interest factors as output data.

[0101] In one embodiment of the present disclosure, the second artificial intelligence model (128) may be a pre-trained eye-focusing intelligence model that receives interest elements and concentration information as input and infers correction weights. The correction weights may include object weights and background weights. The second artificial intelligence model (128) may be a pre-trained artificial intelligence model that infers object weights and correction weights.

[0102] In one embodiment of the present disclosure, a "correction weight" may be a weight used to correct the luminance of an image (200). Specifically, an "object weight" may be a weight used to correct the luminance of at least one object (210) included in an image (200). A "background weight" may be a weight used to correct the luminance of a background (220) included in an image (200).

[0103] At least one processor (130) can provide interest element and concentration information as input data to the second artificial intelligence model (128), thereby obtaining object weights and background weights as output data.

[0104] In one embodiment of the present disclosure, the second artificial intelligence model (128) may be a pre-trained artificial intelligence model that receives interest element and concentration information as input and infers object weights, background weights, and image weights. The "image weights" may be weights used to correct the luminance of the entire image (200).

[0105] At least one processor (130) can provide interest element and concentration information as input data to the second artificial intelligence model (128), thereby obtaining object weights, background weights, and image weights as output data.

[0106] Below, the first artificial intelligence model (127) and the second artificial intelligence model (128) will be described later with reference to FIGS. 8 to 12.

[0107] In one embodiment of the present disclosure, the image correction module (129) may include commands or program codes for obtaining a corrected image (230) in which the brightness of the image (200) is corrected based on correction weights. The image correction module (129) may include commands or program codes for obtaining a corrected image (230) in which the brightness of the image (200) is corrected based on object weights and background weights. The image correction module (129) may include commands or program codes for obtaining a corrected image (230) by correcting R, G, B data of the image (200) or brightness data of the image (200). At this time, the degree of correction for the brightness of at least one object (210) included in the image (200) and the degree of correction for the brightness of the background (220) may be made different by using the object weights and the background weights.

[0108] Additionally, the image correction module (129) may include commands or program codes for obtaining a corrected image (230) in which the brightness of the image (200) is corrected based on the object weight, the background weight, and the image weight.

[0109] The present disclosure is not limited thereto, and the image correction module (129) may include commands or program codes for correcting the brightness of at least one object (210) based on a first correction coefficient and an object weight preset to correspond to at least one object (210), and correcting the brightness of the background (220) based on a second correction coefficient and a background weight preset to correspond to the background (220), thereby obtaining a corrected image (230).

[0110] In one embodiment of the present disclosure, at least one processor (130) may be configured as one or more processors, which control a series of processes to cause the electronic device (100) to operate according to the embodiments described below.

[0111] In one embodiment of the present disclosure, at least one processor (130) may be configured as at least one of a Central Processing Unit, a microprocessor, a Graphic Processing Unit, an Application Processor (AP), an Application Specific Integrated Circuits (ASICs), a Digital Signal Processor (DSPs), a Digital Signal Processing Device (DSPDs), a Programmable Logic Device (PLDs), a Field Programmable Gate Array (FPGAs), a Communication Processor (CP), a Neural Processing Unit, or an artificial intelligence (AI) processor designed with a hardware structure specialized for learning and processing an artificial intelligence (AI) model, but is not limited thereto.

[0112] In one embodiment of the present disclosure, if one or more processors included in at least one processor (130) are artificial intelligence-dedicated processors, the artificial intelligence-dedicated processor may be designed with a hardware structure specialized for processing a specific artificial intelligence model.

[0113] In one embodiment of the present disclosure, at least one processor (130) may be configured as a circuit, such as a System on Chip (SoC) or an Integrated Circuit (IC). At least one processor (130) may include a processing circuit.

[0114] In one embodiment of the present disclosure, at least one processor (130) can execute various types of modules stored in the memory (120). At least one processor (130) can execute at least one instruction constituting the various types of modules stored in the memory (120). By executing the program or at least one instruction stored in the memory (120), at least one processor (130) can process data according to predefined operation rules or artificial intelligence models.

[0115] In one embodiment of the present disclosure, at least one processor (130) may include multiple processors. In one embodiment of the present disclosure, at least one module among the multiple modules in the memory (120) may be executed by any one of the multiple processors. The remaining modules among the multiple modules stored in the memory (120) may be executed by other processors among the multiple processors.

[0116] In one embodiment of the present disclosure, the camera (140) can capture a user image by capturing a physical environment space around the electronic device (100) or at least one user (300) viewing an image (200) through the electronic device (100). In one embodiment of the present disclosure, the camera (140) can include an RGB camera capable of capturing an image including RGB information. However, the present disclosure is not limited thereto, and the camera (140) can include a stereo camera including two RGB cameras, an RGB-Depth camera that captures an image including RGB information and depth information, or a black-and-white camera that captures a black-and-white image, but is not limited to any one of them.

[0117] At least one processor (130) can acquire an image of at least one user (300) viewing the video (200) through a camera (140).

[0118] In one embodiment of the present disclosure, the illuminance sensor (150) can measure the illuminance of a space in which the electronic device (100) is located. In one embodiment of the present disclosure, the illuminance sensor (150) can include a photodiode, a phototransistor, or the like. However, the present disclosure is not limited thereto, and the illuminance sensor (150) can of course include various configurations for measuring the illuminance of a space in which the electronic device (100) is included.

[0119] At least one processor (130) can obtain illumination information of the electronic device (100) through the illumination sensor (150). At this time, the illumination information of the electronic device (100) may mean information about the illumination around the electronic device (100).

[0120] In one embodiment of the present disclosure, the input / output interface (160) can receive at least one of image data or audio data corresponding to an image from an external electronic device or the like under the control of at least one processor (130).

[0121] Additionally, the input / output interface (160) may provide a luminance-corrected corrected image (230) to an external electronic device including a display or an external server under the control of at least one processor (130).

[0122] In one embodiment of the present disclosure, the input / output interface (160) can perform input / output operations with an external electronic device using at least one of input / output methods including an HDMI port (High-Definition Multimedia Interface port), a DVI (Digital Visual Interface), a component jack, a PC port, or a USB port (Universal Serial Bus port). However, the present disclosure is not limited to the above-described input / output methods.

[0123] In one embodiment of the present disclosure, the communication interface (170) can perform data communication with an external server or external electronic device under the control of at least one processor (130).

[0124] The communication interface (170) may perform data communication with an external server or an external electronic device using at least one of data communication methods including, for example, wired LAN, wireless LAN, Wi-Fi, Bluetooth, zigbee, WFD (Wi-Fi Direct), infrared communication (IrDA, infrared Data Association), BLE (Bluetooth Low Energy), NFC (Near Field Communication), Wireless Broadband Internet (Wibro), World Interoperability for Microwave Access (WiMAX), Shared Wireless Access Protocol (SWAP), Wireless Gigabit Alliance (WiGig), and RF communication.

[0125] FIG. 3 is a flowchart illustrating the operation of an electronic device according to one embodiment of the present disclosure.

[0126] Referring to FIGS. 1, 2, and 3, in one embodiment of the present disclosure, a method of operating an electronic device (100) may include a step (S1000) of determining an element of interest of a user among at least one object (210) and at least one background (220) based on segmentation information based on at least one object (21) and a background (220) included in an image (200) and gaze information acquired through a camera (140).

[0127] In the step (S1000) of determining at least one element of interest of a user, at least one processor (130) executes instructions or program codes included in the memory (120), so that the electronic device (100) can determine at least one element of interest of the user among at least one object (210) and at least one background (220) based on segmentation information based on at least one object (21) and background (220) included in the image (200) and gaze information acquired through the camera (140).

[0128] In one embodiment of the present disclosure, step S1000 may include steps S100, S200, and S300. The operations in step S1000 may be performed separately in steps S100, S200, and S300. However, the present disclosure is not limited thereto, and it goes without saying that the operations performed in steps S100, S200, and S300 may also be performed in a single step S1000.

[0129] In one embodiment of the present disclosure, step S1000 may include a step (S100) of segmenting at least one object (210) and a background (220) included in an image (200) to obtain segmentation information.

[0130] In the step (S100) of obtaining segmentation information, at least one processor (130) can obtain segmentation information by executing commands or program codes of the image segmentation module (121) to segment at least one object (210) and background (220) included in the image (200). At this time, the image (200) mentioned in step S100 and step S400 to be described later may mean image data obtained from an external server or electronic device, or stored in a memory (120).

[0131] In one embodiment of the present disclosure, step S1000 may include a step (S200) of obtaining gaze information of at least one user (300) based on a facial image (610, see FIG. 6) of at least one user obtained through a camera (140).

[0132] In the step (S200) of obtaining gaze information of at least one user (300), at least one processor (130) can obtain gaze information of at least one user (300) based on a facial image (610) by executing commands or program codes of the gaze tracking module (123).

[0133] In one embodiment of the present disclosure, the operating method of the electronic device (100) may further include a step of capturing at least one user (300) through a camera (140) to obtain a user image (600, see FIG. 6). In addition, the operating method of the electronic device (100) may further include a step of recognizing a face of at least one user (300) from the obtained user image to obtain a face image (610) of at least one user. In this case, the step of obtaining the user image (600) and the step of obtaining the face image (610) of the user may be performed before the step (S200) of obtaining gaze information of the user (300).

[0134] However, the present disclosure is not limited thereto, and the operation of acquiring a user image (600), the operation of acquiring a user's face image (610), and the operation of acquiring gaze information of the user (300) may be performed in one step.

[0135] In one embodiment of the present disclosure, step S1000 may include a step (S300) of determining an element of interest of at least one user (300) among at least one object (210) and at least one background (220) based on segmentation information and gaze information.

[0136] In the step (S300) of determining an element of interest of at least one user (300), at least one processor (130) may execute instructions or program codes of an element of interest determination module (124) to determine an element of interest of at least one user (300) among at least one object (210) and a background (220) based on segmentation information and gaze information. In one embodiment of the present disclosure, an object overlapping with an area where the gaze of at least one user (300) is located among a plurality of objects representing at least one object (210) and a background (220) segmented within an image (200) may be determined as an element of interest.

[0137] In one embodiment of the present disclosure, an operating method of an electronic device (100) may include a step (S2000) of obtaining concentration information corresponding to at least one element of interest through a first artificial intelligence model based on at least one user's face image (610) obtained through a camera (140), brightness information of an image (200), and illumination information obtained through an illumination sensor (150).

[0138] In the step (S2000) of obtaining concentration information corresponding to at least one element of interest, at least one processor (130) executes commands or program codes included in the memory (120), so that the electronic device (100) can obtain concentration information corresponding to at least one element of interest through the first artificial intelligence model based on at least one user's face image (610) obtained through the camera (140), brightness information of the image (200), and illumination information obtained through the illumination sensor (150).

[0139] In one embodiment of the present disclosure, step S2000 may include steps S400, S500, and S600. The operations in step S2000 may be performed separately in steps S400, S500, and S600. However, the present disclosure is not limited thereto, and it goes without saying that the operations performed in steps S400, S500, and S600 may be performed in a single step S2000. In one embodiment of the present disclosure, step S2000 may include a step (S400) of acquiring brightness information of an image (200) based on the image (200).

[0140] In the step (S400) of acquiring brightness information of an image (200), at least one processor (130) can acquire brightness information of an image (200) by executing commands or program codes of a brightness information acquisition module (125).

[0141] In one embodiment of the present disclosure, step S2000 may include a step (S500) of obtaining illumination information of the electronic device (100) through an illumination sensor (150).

[0142] In the step (S500) of acquiring illumination information of an electronic device (100), at least one processor (130) can acquire illumination information of the electronic device (100) through an illumination sensor (150) by executing commands or program codes of an illumination acquisition module (126).

[0143] In one embodiment of the present disclosure, step S2000 may include a step S600 of obtaining concentration information of at least one user (300) through a first artificial intelligence model based on a facial image of at least one user (300), brightness information, and illuminance information of the electronic device (100). At this time, since an element of interest of at least one user (300) is determined based on the gaze information and segmentation information of the gaze of at least one user (300) in step S300, the concentration information obtained in step S600 may mean concentration information on the element of interest.

[0144] In the step (S600) of obtaining concentration information on an element of interest of at least one user (300), at least one processor (130) may provide a facial image, brightness information, and illuminance information of the electronic device (100) of at least one user (300) as inputs to a first artificial intelligence model, thereby obtaining concentration information of at least one user (300).

[0145] In one embodiment of the present disclosure, the operating method of the electronic device (100) may include a step (S3000) of obtaining a corrected image (230) in which the brightness of the image (200) is corrected according to a correction weight obtained through a second artificial intelligence model based on information on elements of interest and concentration.

[0146] In the step (S3000) of obtaining a corrected image (230) in which the brightness of the image (200) is corrected according to the acquired correction weight, at least one processor (130) executes commands or program codes included in the memory (120), so that the electronic device (100) can obtain a corrected image (230) in which the brightness of the image (200) is corrected according to the correction weights acquired through the second artificial intelligence model based on the interest element and concentration information.

[0147] In one embodiment of the present disclosure, step S3000 may include steps S700 and S800. The operations in step S3000 may be performed separately in steps S700 and S800. However, the present disclosure is not limited thereto, and it goes without saying that the operations performed in steps S700 and S800 may also be performed in a single step S3000.

[0148] In one embodiment of the present disclosure, step S3000 may include a step (S700) of obtaining object weights and background weights through a second artificial intelligence model based on interest elements and concentration information.

[0149] In the step of obtaining object weights and background weights (S700), at least one processor (130) can provide interest element and concentration information as inputs to the second artificial intelligence model to obtain object weights and background weights.

[0150] In one embodiment of the present disclosure, step S3000 may include a step (S800) of obtaining a corrected image (230) in which the brightness of the image (200) is corrected based on the object weight and the background weight.

[0151] In the step (S800) of obtaining a corrected image (230) in which the brightness of the image (200) is corrected, at least one processor (130) may execute commands or program codes of the image correction module (129) to correct the brightness of at least one object (210) based on the object weight and to correct the brightness of the background (220) based on the background weight, thereby obtaining a corrected image (230).

[0152] Although FIG. 3 illustrates multiple operations being performed separately, the present disclosure is not limited thereto. In the operation of the electronic device (100), some of the multiple operations illustrated in FIG. 3 may be omitted, or new operations may be added. Furthermore, it goes without saying that two or more of the multiple operations illustrated in FIG. 3 may be performed together in a single step.

[0153] FIG. 4 is a diagram for explaining the operation of an electronic device according to an embodiment of the present disclosure. FIG. 5 is a diagram for explaining the operation of segmenting an object and a background included in an image and obtaining segmentation information according to an embodiment of the present disclosure. FIG. 6 is a diagram for explaining the operation of acquiring a user's face image and user's gaze information according to an embodiment of the present disclosure. FIG. 7 is a diagram for explaining the operation of acquiring illumination information of an electronic device according to an embodiment of the present disclosure.

[0154] Hereinafter, for convenience of explanation, it is described that the image (400) includes multiple objects and multiple backgrounds, and multiple users use the electronic device (100).

[0155] Referring to FIGS. 1, 2, and 4, in one embodiment of the present disclosure, an electronic device (100) may acquire an image (400). At this time, the image (400) acquired by the electronic device (100) may include a plurality of objects (411) and a plurality of backgrounds (416).

[0156] In one embodiment of the present disclosure, the electronic device (100) can obtain segmentation information by segmenting a plurality of objects (411) and a plurality of backgrounds (416) included in the image (400) based on the image (400).

[0157] Referring to FIGS. 4 and 5, in one embodiment of the present disclosure, FIG. 5 illustrates an image (400) including a plurality of objects (411) and a plurality of backgrounds (416).

[0158] In one embodiment of the present disclosure, the plurality of objects (411) may include a first object (412), a second object (413), a third object (414), and a fourth object (415). The plurality of backgrounds (416) may include a first background (417), a second background (418), and a third background (419). However, the present disclosure is not limited thereto, and the image (400) may of course include more or fewer objects or backgrounds.

[0159] In one embodiment of the present disclosure, the electronic device (100) can obtain segmentation information for segmenting the first to fourth objects (412, 413, 414, 415) and the first to third backgrounds (417, 418, 419) included in the image (400). At this time, the segmentation information can include location, size, or classification information of the first to fourth objects (412, 413, 414, 415) and the first to third backgrounds (417, 418, 419).

[0160] In one embodiment of the present disclosure, each object included in an image (400) segmented by an electronic device (100) may be referred to as an “image element.” In this case, each object may include at least first to fourth objects (412, 413, 414, 415) and first to third backgrounds (417, 418, 419).

[0161] Referring back to FIG. 4, in one embodiment of the present disclosure, the electronic device (100) can capture a plurality of users via a camera (140) to obtain a user image (430). The electronic device (100) can recognize the faces of the plurality of users from the user image (430) to obtain a user face image (440). In this case, the user face image (440) can include face images of each of the plurality of users.

[0162] In one embodiment of the present disclosure, the electronic device (100) can obtain gaze information (450) of a plurality of users based on a user face image (440). The electronic device (100) can detect eye positions and facial landmarks of the plurality of users from the user face image (400), and obtain gaze information (450) of the plurality of users based on the eye positions and landmarks. At this time, the gaze information (450) can include information on the gaze positions of each of the plurality of users.

[0163] Referring to FIGS. 4 and 6, in one embodiment of the present disclosure, FIG. 6 illustrates a user image (600) obtained by photographing three users and a user face image (610) obtained by recognizing the faces of the three users from the user images (600).

[0164] In one embodiment of the present disclosure, the electronic device (100) can detect facial landmarks and pupil locations included in each face from user face images (610) in which the faces of three users are recognized. In this case, the method for detecting facial landmarks and pupil locations is a commonly used technique, and thus a detailed description thereof will be omitted.

[0165] In one embodiment of the present disclosure, the electronic device (100) can acquire gaze information of three users based on the detected facial landmarks and pupil locations. At this time, the gazes of the three users may be located in different areas within the image (400).

[0166] Referring again to FIG. 4, in one embodiment of the present disclosure, multiple users may view the image (200) by focusing their gaze on different objects among multiple objects (411) and multiple backgrounds (412) included in the image (200). The electronic device (100) may determine the interest elements (420) of each of the multiple users based on segmentation information and gaze information (450).

[0167] In one embodiment of the present disclosure, the electronic device (100) can obtain image brightness information (461) from the acquired image (400). At this time, the image brightness information (461) can include image luminance information obtained from the grayscale data or brightness data of the image (400).

[0168] In one embodiment of the present disclosure, the electronic device (100) can obtain illumination information (462) of the electronic device (100) through the illumination sensor (150). The illumination information (462) may refer to the illumination of a space where the electronic device (100) is located or the illumination of the surroundings of the electronic device (100) that may affect viewing of an image displayed through the electronic device (100).

[0169] Referring to FIGS. 4 and 7, in one embodiment of the present disclosure, a light (700) may be positioned around an electronic device (100) displaying an image (400). At this time, the illumination information of the electronic device (100) may be changed by the brightness of the light (700). However, the present disclosure is not limited thereto. It goes without saying that the illumination information of the electronic device (100) may be changed not only by the light (700) but also by other electronic devices or sunlight, etc. around the electronic device (100). Accordingly, the concentration of multiple users viewing the image (400) may be affected.

[0170] Specifically, if the illumination information of the electronic device (100) is excessively brighter than the brightness of the image (400), the visibility of the image (400) may decrease, thereby lowering the concentration of multiple users. In addition, if the illumination information of the electronic device (100) is darker than the brightness of the image (400), the visibility of the image (400) may increase, thereby increasing the concentration of multiple users. However, this is merely an example, and the present disclosure is not limited thereto. Depending on the relative relationship between the brightness of the image (400) and the illumination information of the electronic device (100), the concentration of multiple users may be differently affected.

[0171] The electronic device (100) can obtain the concentration of multiple users corresponding to the brightness information and illuminance information of the image (400) through a pre-trained second artificial intelligence model.

[0172] Referring back to FIG. 4, in one embodiment of the present disclosure, the electronic device (100) may provide a user face image (440), brightness information (461) and illuminance information (462) of the image to the first artificial intelligence model (127) to obtain user concentration information (470). The user concentration information (470) may include concentration information on elements of interest of each of a plurality of users. In one embodiment of the present disclosure, the user concentration information (470) may have a higher value as the user's concentration on the element of interest increases, and may have a lower value as the user's concentration on the element of interest decreases.

[0173] In one embodiment of the present disclosure, the electronic device (100) may provide interest elements (420) and concentration information (470) to the second artificial intelligence model (128), which may include object weights and background weights (480).

[0174] At this time, the object weight may include a plurality of sub-object weights corresponding to a plurality of objects (411). At least one sub-object weight among the plurality of sub-object weights may have a different value. Specifically, the values ​​of the plurality of sub-object weights determined by the second artificial intelligence model (128) to maximize the concentration of all users may be obtained differently depending on whether the weight is determined as an element of interest of each of the plurality of users and the concentration of each of the plurality of users.

[0175] Additionally, the background weight may include a plurality of sub-background weights corresponding to a plurality of backgrounds (412). At least one sub-background weight among the plurality of sub-background weights may have a different value. Specifically, the values ​​of the plurality of sub-background weights determined by the second artificial intelligence model (128) to maximize the concentration of all users may be obtained differently depending on whether the plurality of users have each determined the element of interest and the concentration of each of the plurality of users.

[0176] In one embodiment of the present disclosure, the electronic device (100) can obtain a corrected image (490) that corrects the brightness of the image (400) using object weights and background weights (480). The electronic device (100) can correct the brightness of a plurality of objects (411) included in the image (400) based on the object weights, and can correct the brightness of a plurality of backgrounds (412) included in the image (400) based on the background weights to generate the corrected image (490).

[0177] In one embodiment of the present disclosure, the electronic device (100) can display the acquired corrected image (490) via the display (110) and provide it to multiple users. The corrected image (490) is an image whose brightness has been corrected based on object weights and background weights acquired through a pre-trained second artificial intelligence model to increase the combined concentration of multiple users. Accordingly, the electronic device (100) can increase the concentration of multiple users by providing the corrected image (490).

[0178] FIG. 8 is a diagram illustrating an operation for obtaining concentration information on a user's interest element through a first artificial intelligence model according to one embodiment of the present disclosure. FIG. 9 is a flowchart illustrating an operation for obtaining a transformed face image according to one embodiment of the present disclosure.

[0179] Referring to FIGS. 2, 4, and 8, in one embodiment of the present disclosure, FIG. 8 illustrates a first artificial intelligence model (127). In one embodiment of the present disclosure, the first artificial intelligence model (127) may include a plurality of neural network layers. Each of the plurality of neural network layers may include a plurality of weights. The first artificial intelligence model (127) may include, but is not limited to, a convolutional neural network (CNN), a recurrent neural network (RNN), a transformer model, and the like.

[0180] In one embodiment of the present disclosure, the first artificial intelligence model (127) may be a pre-trained artificial intelligence model that receives as input at least one user's face image (800), brightness information (820) of an image (400), and illuminance information (830) of an electronic device (100) using a training data set to infer concentration information (850).

[0181] In one embodiment of the present disclosure, the first artificial intelligence model (127) may be an artificial intelligence model pre-trained by a training data set including brightness of a training image, illumination information of an electronic device providing the training image, at least one facial image viewing the training image, and concentration information obtained from at least one user viewing the training image. In this case, the concentration information may be measured concentration information obtained by attaching electrodes capable of measuring brain waves to at least one user viewing the training image and analyzing the measured brain waves.

[0182] In one embodiment of the present disclosure, the first artificial intelligence model (127) may be pre-learned through a process of updating a plurality of weights using a loss function based on the brightness of the training image, the illumination information of the electronic device providing the training image, and the concentration information obtained based on at least one face image viewing the training image, and the measured concentration information obtained by analyzing brain waves.

[0183] In one embodiment of the present disclosure, the electronic device (100) can obtain at least one user's concentration information (850) based on the user's face image (800), brightness information (820), and illuminance information (830) by using a first artificial intelligence model (127) including a plurality of weights that are pre-learned and optimized.

[0184] In one embodiment of the present disclosure, the electronic device (100) may perform preprocessing on at least one of the user's face image (800), brightness information (820), or illuminance information (830) to fit the first artificial intelligence model (127). The preprocessing of the data may be to convert the data into a form understandable to the artificial intelligence model or to improve the quality of the data to enhance the effectiveness of learning by utilizing high-level learning data.

[0185] Referring to FIGS. 8 and 9 , in one embodiment of the present disclosure, the method of operating the electronic device (100) may further include a step (S510) of converting a face image (800) into a black and white image to obtain a converted face image (810). In one embodiment of the present disclosure, step S510 may be performed after step S500. However, the present disclosure is not limited thereto, and step S510 may also be performed after the step of obtaining the user's face image.

[0186] In one embodiment of the present disclosure, when the camera (140) is an RGB camera, the facial image (800) of at least one user may be an RGB image. At least one processor (130) may convert the facial image (800) of at least one user into a black-and-white image by executing a command of a model that converts a three-dimensional RGB image into a two-dimensional black-and-white image, thereby obtaining a converted facial image (810).

[0187] In one embodiment of the present disclosure, the electronic device (100) can convert at least one user's face image (800), which is an RGB image, into a black and white image to obtain a converted face image (810).

[0188] In one embodiment of the present disclosure, in the step (S600) of obtaining concentration information of at least one user (300), the electronic device (100) may provide a converted face image (810), brightness information (820), and illuminance information (830) as inputs to the first artificial intelligence model (127), thereby obtaining concentration information (850). In the step S600, at least one processor (130) may provide the converted face image (810), brightness information (820), and illuminance information (830) as inputs to the first artificial intelligence model (127), thereby obtaining concentration information (850).

[0189] However, the present disclosure is not limited thereto, and the electronic device (100) may also perform preprocessing on brightness information (820) and illuminance information (830).

[0190] In one embodiment of the present disclosure, the brightness information (820) and the illuminance information (830) may each be two-dimensional data. In one embodiment of the present disclosure, the electronic device (100) may pad at least one of the brightness information (820) and the illuminance information (830) to match the size of the brightness information (820) and the size of the illuminance information (830) with the size of the converted face image (810). In addition, the electronic device (100) may also perform normalization on at least one of the brightness information (820) and the illuminance information (830).

[0191] However, the present disclosure is not limited thereto, and the electronic device (100) can perform various types of preprocessing on at least one of the user's face image (800), brightness information (820), or illuminance information (830) to suit the first artificial intelligence model (127).

[0192] In one embodiment of the present disclosure, the electronic device (100) can concatenate a preprocessed transformed face image (810), brightness information (820), and illuminance information (830) to generate three-dimensional input data (840) suitable for the first artificial intelligence model (127). In one embodiment of the present disclosure, the electronic device (100) can provide the three-dimensional input data (840) as input data to the first artificial intelligence model (127).

[0193] However, the present disclosure is not limited thereto, and it goes without saying that the size or dimension of input data provided by the electronic device (100) to the first artificial intelligence model (127) may be changed depending on the type or structure of the first artificial intelligence model (127).

[0194] In one embodiment of the present disclosure, the concentration information (850) obtained through the first artificial intelligence model (127) may be a normalized value. When the concentration of at least one user is inferred to be high through the pre-trained first artificial intelligence model (127), the concentration information (850) may have a value close to 1, and when the concentration is inferred to be low, the concentration information (850) may have a value close to 0.

[0195] FIG. 10 is a diagram illustrating an operation of obtaining object weights and background weights through a second artificial intelligence model according to one embodiment of the present disclosure. FIG. 11 is a flowchart illustrating an operation of obtaining image weights through a second artificial intelligence model and obtaining a corrected image based on the object weights, background weights, and image weights according to one embodiment of the present disclosure.

[0196] Referring to FIGS. 2, 3, 4, and 10, in one embodiment of the present disclosure, FIG. 10 illustrates a second artificial intelligence model (128). In one embodiment of the present disclosure, the second artificial intelligence model (128) may include a plurality of neural network layers. Each of the plurality of neural network layers may include a plurality of weights. The second artificial intelligence model (128) may include, but is not limited to, a convolutional neural network (CNN), a recurrent neural network (RNN), a transformer model, and the like.

[0197] In step S700, the second artificial intelligence model (128) may be an artificial intelligence model that has been pre-trained through reinforcement learning using rewards to infer object weights and background weights (1010) by receiving interest elements (420) and concentration information (470) as inputs.

[0198] In one embodiment of the present disclosure, the reward may include the difference between first concentration information obtained from at least one user viewing the training video and second concentration information obtained from at least one user viewing a training correction image in which the brightness of the training video is corrected based on object weights and background weights obtained through a second artificial intelligence model (128). In this case, the first concentration information and the second concentration information may be measured information obtained by attaching electrodes capable of measuring brain waves to at least one user viewing the training video and the training correction image and analyzing the measured brain waves.

[0199] In one embodiment of the present disclosure, in step S700, the electronic device (100) can obtain object weights and background weights (1010) based on the interest factor (420) and concentration information (470) using a second artificial intelligence model (128) including a plurality of weights that have been pre-learned and optimized.

[0200] In one embodiment of the present disclosure, the object weight and the background weight (1010) may each have values ​​from 0 to 1.

[0201] In one embodiment of the present disclosure, when a plurality of objects are included in an image (200), the object weight may include a plurality of sub-object weights corresponding to each of the plurality of objects. When a plurality of backgrounds are included in an image (200), the background weight may include a plurality of sub-background weights corresponding to each of the plurality of backgrounds. Each of the plurality of sub-object weights and the plurality of sub-background weights may have a value from 0 to 1.

[0202] In step S800, the electronic device (100) can correct the brightness of the image (200, see FIG. 1) using the mathematical expression (1) below.

[0203] Mathematical formula 1:

[0204] At this time, if at least one object (210) and background (220) included in the image (200) are divided into n image elements, can be any natural number between 1 and n, Is is the degree of luminance correction of the th image element, Is is the correction coefficient of the th image element, Is is the weight of the th image element, may be a preset correction constant.

[0205] In one embodiment of the present disclosure, n may be the sum of the number of multiple objects and the number of multiple backgrounds. The second image element may refer to any one of a plurality of objects and a plurality of backgrounds. Among the multiple sub-object weights and multiple sub-background weights obtained through the second artificial intelligence model (128), It can mean the weight corresponding to the th image element.

[0206] The larger the weight value of the th image element, The degree of luminance correction of the second image element can be increased. The smaller the weight of the th image element, the The degree of luminance correction of the second image element may be reduced.

[0207] In one embodiment of the present disclosure, is one of several predetermined correction factors. It may be a correction factor determined depending on the type of the second video element.

[0208] In one embodiment of the present disclosure, a plurality of predetermined correction coefficients are a first correction coefficient preset to correspond to an object. and the second correction coefficient preset to correspond to the background. may include. If the th image element is an object, can be determined as the first correction coefficient. If the th video element is the background, can be determined by the second correction coefficient.

[0209] At this time, the magnitude of the first correction coefficient and the magnitude of the second correction coefficient may be different from each other. Specifically, the magnitude of the first correction coefficient may be greater than the magnitude of the second correction coefficient. Accordingly, the degree of luminance correction when the component whose luminance is to be corrected is an object may be greater than the degree of luminance correction when the component whose luminance is to be corrected is a background.

[0210] Since a user watching a video tends to focus his / her gaze on an object rather than the background included in the video, the user's concentration on the electronic device (100) can be improved by adjusting the luminance correction level of the object to be higher than that of the background.

[0211] However, the present disclosure is not limited thereto, and it goes without saying that the electronic device (100) can also correct the brightness of the image (200) using a mathematical formula in which the correction coefficient is excluded from mathematical formula (1).

[0212] In one embodiment of the present disclosure, The electronic device (100) is included in the image (200). This may refer to the degree to which the luminance of the second image element is corrected. is included in the image before the electronic device (100) corrects the brightness. The luminance of the th image element and the image after luminance correction. This may mean the difference in luminance of the second image element.

[0213] In one embodiment of the present disclosure, the electronic device (100) can correct the brightness of an image (200) using n brightness correction degrees calculated through mathematical expression (1) from the first image element to the nth image element. Accordingly, an optimized corrected image (230) can be obtained to improve the concentration of at least one user (300) viewing the image (200).

[0214] Referring to FIGS. 10 and 11, in one embodiment of the present disclosure, the method of operating the electronic device (100) may further include a step (S710) of receiving an element of interest (420) and concentration information (470) as input and obtaining an image weight (1020) for correcting the brightness of the entire image through a second artificial intelligence model (128).

[0215] In step S710, the electronic device (100) can obtain image weights (1020) based on the interest elements (420) and concentration information (470) using a second artificial intelligence model (128) including a plurality of weights that have been pre-learned and optimized.

[0216] At this time, step S710 may be performed after step S700. However, the present disclosure is not limited thereto, and the operation of obtaining object weights and background weights (1010) and the operation of obtaining image weights (1020) through the second artificial intelligence model (128) may be performed in a single step. The object weights and background weights (1010) and the image weights (1020) may be obtained as output by providing interest elements (420) and concentration information (470) as inputs to the second artificial intelligence model (128).

[0217] In one embodiment of the present disclosure, the second artificial intelligence model (128) may be an artificial intelligence model that has been pre-trained through reinforcement learning using rewards to receive interest elements (420) and concentration information (470) as inputs and infer object weights and background weights (1010) and image weights (1020) for correcting the brightness of the entire image.

[0218] In one embodiment of the present disclosure, the reward may include the difference between first concentration information obtained from at least one user viewing the training video and second concentration information obtained from at least one user viewing the training corrected video in which the luminance of the training video is corrected based on the object weights, background weights, and image weights obtained through the second artificial intelligence model (128).

[0219] In one embodiment of the present disclosure, the electronic device (100) can obtain object weights, background weights (1010), and image weights (1020) based on interest factors (420) and concentration information (470) using a second artificial intelligence model (128) including a plurality of weights that are pre-learned and optimized.

[0220] In one embodiment of the present disclosure, the operating method of the electronic device (100) may further include a step (S810) of obtaining a corrected image (230) based on an object weight, a background weight, and an image weight (1020). Step S810 may be performed after step S710.

[0221] In step S810, the electronic device (100) can correct the brightness of the image (200) using the mathematical expression (2) below.

[0222] Mathematical formula 2:

[0223] At this time, can be any natural number between 1 and n, Is is the degree of luminance correction of the th image element, Is is the correction coefficient of the th image element, Is is the weight of the th image element, is a preset correction constant, can be image weights.

[0224] In one embodiment of the present disclosure, may be a weight for correcting the brightness of the entire image (200), obtained through the second artificial intelligence model (128). As an image weight having a high value is inferred to increase the user's concentration, the electronic device (100) may apply an image weight having a high value when correcting the brightness of n image elements, thereby increasing the degree of brightness correction of the entire image (200).

[0225] As an image weight having a low value is inferred to increase the user's concentration, the electronic device (100) can apply an image weight having a low value when correcting the brightness of n image elements, thereby lowering the brightness correction degree of the entire image (200). At this time, whether an image weight having a high value or an image weight having a low value is necessary to increase the user's (200) concentration can be inferred by the pre-learned second artificial intelligence model (128).

[0226] In one embodiment of the present disclosure, the electronic device (100) may provide graph information (1000) obtained by preprocessing segmentation information, interest elements (420), and concentration information (470) as input data to the second artificial intelligence model (128). The second artificial intelligence model (128) may be pre-trained to infer object weights, background weights (1010), and image weights (1020) based on input data in the form of graphs.

[0227] Below, the graph information (1000) will be described later in Fig. 12.

[0228] FIG. 12 is a diagram for explaining a plurality of nodes and a plurality of edges included in graph information according to one embodiment of the present disclosure.

[0229] Referring to FIGS. 2, 3, 4, 10, and 12, in one embodiment of the present disclosure, a method of operating an electronic device (100) may further include a step of obtaining graph information (1000) including a plurality of nodes and a plurality of edges representing relationships between the plurality of nodes, based on segmentation information, elements of interest (420), and concentration information (470).

[0230] In one embodiment of the present disclosure, the step of obtaining graph information (1000) may be performed after step S600. The electronic device (100) may obtain graph information (1000) including a plurality of nodes and a plurality of edges representing relationships between the plurality of nodes, based on segmentation information, interest elements (420), and concentration information (470).

[0231] In the step of acquiring graph information (1000), the electronic device (100) may acquire graph information (1000) including a plurality of nodes and a plurality of edges representing relationships between the plurality of nodes based on segmentation information, interest elements (420), and concentration information (470). At this time, each of the plurality of edges may include an edge weight.

[0232] In one embodiment of the present disclosure, the plurality of nodes may include a plurality of image element nodes, each corresponding to a plurality of image elements segmented from the image (400, see FIG. 5 ). Additionally, the plurality of nodes may include at least one user node corresponding to at least one user.

[0233] In one embodiment of the present disclosure, the plurality of edges may include at least one user edge connecting user nodes and image element nodes. The plurality of edges may include at least one area edge connecting an object element node corresponding to an object and a background element node corresponding to a background among the image element nodes.

[0234] In one embodiment of the present disclosure, at least one user edge may be an edge connected from a user node corresponding to the user to an image element node corresponding to an image element being viewed by the user, based on the gaze position of at least one user. In this case, the image element node connected to the user edge may be a node corresponding to an image element determined to be a component of interest to the user. In addition, the edge weight of the user edge may indicate information about the user's concentration level.

[0235] In one embodiment of the present disclosure, at least one region edge may be an edge connected from an object node corresponding to an object to a background node corresponding to the background in which the object is located, depending on the positions of the object and the background within the image (200). In this case, the edge weight of the region edge may be a ratio of the degree to which the object overlaps the background. In the case where the object overlaps two or more backgrounds, region edges may be connected from one object node to two or more background nodes, respectively.

[0236] In one embodiment of the present disclosure, the graph information (1000) illustrated in FIG. 12 includes nine nodes and eight edges. The nine nodes include three user nodes, four object nodes, and two background nodes. The eight edges include three user edges and five area edges.

[0237] In one embodiment of the present disclosure, a first user node (1200) corresponding to a first user is connected to a first user edge (1201) that points from the first user node (1200) to a first object node (1220) corresponding to a first object. At this time, the first user edge (1201) may have a first edge weight (1202). The first edge weight (1202), "0.81," may indicate the first user's concentration level information. The first object may be an element of interest of the first user.

[0238] In one embodiment of the present disclosure, a second user node (1210) corresponding to a second user is connected to a second user edge (1211) from the second user node (1210) toward a first object node (1220). At this time, the second user edge (1211) may have a second edge weight (1212). The second edge weight (1212), "0.14," may indicate the second user's concentration level information. The first object may be an element of interest for the second user.

[0239] In one embodiment of the present disclosure, a first object node (1220) is connected to a third user edge (1221) that points from the first object node (1220) to a first background node (1230) corresponding to the first background. At this time, the third user edge (1221) may have a third edge weight (1222). The third edge weight (1222), "1.0", may be a ratio of the first object overlapping the first background. The first object may be located within the first background.

[0240] The electronic device (100) can generate graph information (1000) including types of a plurality of image elements including the above information, positional relationships, interest elements and concentration information of at least one user, etc., based on segmentation information, interest elements (420) and concentration information (470).

[0241] In one embodiment of the present disclosure, the operation of the electronic device (100) for displaying a corrected image obtained by correcting the brightness of an image may be repeatedly performed according to a preset cycle. The electronic device (100) of the present disclosure may repeat the operation of acquiring an element of interest (420), the operation of acquiring gaze information (450), and the operation of acquiring concentration information (470) at preset cycles.

[0242] The electronic device (100) can obtain graph information (1000) in the current cycle based on the interest elements (420) and concentration information (470) obtained in the previous cycle. The electronic device (100) can provide the graph information (1000) in the current cycle to a second artificial intelligence model (128) to provide a corrected image in the current cycle with brightness corrected based on the obtained object weights, background weights, and image weights.

[0243] At this time, during the initial operation of the electronic device (100) (e.g., immediately after turning on), the interest elements (420) and concentration information (470) acquired in the previous cycle may not exist. In this case, the electronic device (100) may generate graph information (1000) using the initial conditions (conditions 1 to 9) below.

[0244] Condition 1:

[0245] Condition 2:

[0246] Condition 3:

[0247] Condition 4:

[0248] Condition 5:

[0249] Condition 6:

[0250] Condition 7:

[0251] Condition 8:

[0252] Condition 9:

[0253] At this time, can mean the weight of an edge going out from a background node. may mean the sum of the weights of at least one edge pointing outward from one object node. may mean the sum of the weights of at least one edge outward from one user node. may mean the number of at least one edge pointing to one background node. may mean the number of at least one edge going out from one background node. may mean the number of at least one edge directed to one object node. may mean the number of at least one edge pointing outward from one object node. may mean the number of at least one edge directed to one user node. may mean the number of at least one edge going out from one user node.

[0254] The electronic device (100) can generate arbitrary graph information (1000) that satisfies the aforementioned initial conditions, and acquire object weights, background weights, and image weights for correcting the brightness of an image during initial operation. Thereafter, the electronic device (100) can correct the brightness of the image using the acquired graph information based on the acquired interest elements (420) and concentration information (470).

[0255] FIG. 13 is a diagram for explaining an operation of training a second artificial intelligence model according to one embodiment of the present disclosure.

[0256] In one embodiment of the present disclosure, the learning of the second artificial intelligence model in FIG. 13 may be performed in the electronic device (100). However, in one embodiment of the present disclosure, the learning of the second artificial intelligence model may be performed in an external server or an external electronic device, and the electronic device (100) may acquire a pre-learned second artificial intelligence model or may acquire and utilize a plurality of optimized weights.

[0257] Referring to FIGS. 2, 10, 12, and 13, in one embodiment of the present disclosure, as the electronic device (100) provides graph information (1300) including a plurality of nodes, a plurality of edges, and a plurality of weights as input data to the second artificial intelligence model (128), object weights and background weights (1310) and image weights (1320) can be obtained as output data. The electronic device (100) can provide a user with a training correction image in which the brightness of the training image is corrected based on the object weights and background weights (1310) and the image weights (1320) obtained through the image correction module (129).

[0258] In one embodiment of the present disclosure, the electronic device (100) can obtain first concentration information from a user viewing a training video without luminance correction. Furthermore, the electronic device (100) can obtain second concentration information from a user viewing a training video with luminance correction. In this case, the first concentration information and the second concentration information may be obtained based on brain waves measured through electrodes that measure the user's brain waves.

[0259] In one embodiment of the present disclosure, the first concentration information and the second concentration information may be normalized values ​​to have values ​​between 0 and 1, respectively. The electronic device (100) may obtain a concentration change amount (1330), which is a difference between the second concentration information and the first concentration information. In one embodiment of the present disclosure, when there are multiple users viewing the training video and the training correction video, each of the first concentration information and the second concentration information may be the sum of the concentrations of each of the multiple users.

[0260] In one embodiment of the present disclosure, the electronic device (100) may provide feedback to the second artificial intelligence model (128) to enable reinforcement learning of the second artificial intelligence model (128) to infer optimal object weights, background weights (1310), and image weights (1320) that maximize the amount of concentration change (1330). In this case, the amount of concentration change (1330) may be used as a reward in reinforcement learning.

[0261] Through the above process, the second artificial intelligence model (128) can be trained to infer optimal object and background weights (1310) and image weights (1320) used to correct the brightness of an image in order to increase the concentration of a user using the electronic device (100). In addition, even when multiple users use the electronic device (100), the second artificial intelligence model (128) can be trained to infer optimal object and background weights (1310) and image weights (1320) by taking into account the concentration of all of the multiple users.

[0262] FIG. 14 is a flowchart for explaining an operation of displaying a correction image through a display according to one embodiment of the present disclosure.

[0263] Referring to FIGS. 1, 2, 3, and 14, in one embodiment of the present disclosure, the method of operating the electronic device (100) may further include a step (S900) of displaying a corrected image (230) through a display (110). In one embodiment of the present disclosure, step S900 may be performed after step S800 of obtaining the corrected image (230).

[0264] In one embodiment of the present disclosure, in the step (S900) of displaying the corrected image (230), at least one processor (130) can control the display (110) to display the corrected image (230).

[0265] In one embodiment of the present disclosure, the electronic device (100) may analyze the gaze information of at least one user (300) using the electronic device (100) or the concentration of at least one user (300), and provide a corrected image (230) with a brightness that is optimized to improve the concentration of at least one user (300) to the user (300).

[0266] At this time, the luminance of the corrected image (230) is not uniformly corrected, but the degree of luminance correction of an element of interest where the gaze of at least one user (300) is located may be greater than the degree of luminance correction of other image elements. In addition, when multiple users (300) use the electronic device (100), the luminance of multiple objects and multiple backgrounds included in the image (200) may be corrected by considering multiple elements of interest according to gaze information of each of the multiple users (300) and the degree of concentration of each of the multiple users (300).

[0267] Accordingly, an image (230) that has been corrected to have optimized brightness by taking into account the usage environment, concentration, status, etc. of at least one user (300) using the electronic device (100) can be provided to at least one user (300).

[0268] To solve the above-described technical problem, in one embodiment of the present disclosure, an electronic device is provided. The electronic device may include an illumination sensor. The electronic device may include a camera. The electronic device may include a memory storing at least one instruction. The electronic device may include at least one processor that executes at least one instruction stored in the memory. By executing at least one instruction by the at least one processor, the electronic device may determine at least one element of interest among at least one object and at least one background based on segmentation information based on at least one object and background included in an image and gaze information acquired through a camera. By executing at least one instruction by the at least one processor, the electronic device may obtain concentration information corresponding to at least one element of interest through a first artificial intelligence model based on at least one face image acquired through the camera, brightness information of the image, and concentration information acquired through an illumination sensor. By executing at least one instruction by the at least one processor, the electronic device may control the luminance of the image to be corrected according to a correction weight acquired through a second artificial intelligence model based on the element of interest and concentration information.

[0269] In one embodiment of the present disclosure, the camera is an RGB camera, and at least one user's face image may be an RGB image. By having at least one processor execute at least one command, the electronic device can convert the face image into a black-and-white image to obtain a converted face image. By having at least one processor execute at least one command, the electronic device can obtain concentration information through a first artificial intelligence model based on the converted face image, brightness information, and illuminance information.

[0270] In one embodiment of the present disclosure, the first artificial intelligence model may be a pre-trained artificial intelligence model that receives a facial image, brightness information, and illuminance information as input using a training data set and infers concentration information. The training data set may include brightness of a training image, illuminance information of an electronic device providing the training image, a facial image of at least one user viewing the training image, and concentration information obtained from at least one user viewing the training image.

[0271] In one embodiment of the present disclosure, by having at least one processor execute at least one command, an electronic device can obtain graph information including a plurality of nodes and a plurality of edges representing relationships between the plurality of nodes based on segmentation information, elements of interest, and concentration information. By having at least one processor execute at least one command, the electronic device can obtain correction weights through a second artificial intelligence model based on the graph information.

[0272] In one embodiment of the present disclosure, the plurality of nodes may include a first node corresponding to at least one object, a second node corresponding to a background, and a third node corresponding to at least one user. The plurality of edges may include a first edge from the first node toward the second node, a first edge having a first edge weight, and a second edge from the third node toward either the first node or the second node, and a second edge having a second edge weight.

[0273] In one embodiment of the present disclosure, the second artificial intelligence model may be a pre-trained artificial intelligence model that receives interest elements and concentration information as input and infers correction weights through reinforcement learning using rewards. The reward may include the difference between first concentration information obtained from at least one user viewing a training video and second concentration information obtained from at least one user viewing a training-corrected video in which the luminance of the training video is corrected.

[0274] In one embodiment of the present disclosure, the correction weights may include an object weight corresponding to the luminance correction of at least one object and a background weight corresponding to the luminance correction of the background. By having at least one processor execute at least one command, the electronic device may obtain a corrected image in which the luminance of the image is corrected based on the object weights and the background weights.

[0275] In one embodiment of the present disclosure, by having at least one processor execute at least one command, the electronic device can correct the luminance of at least one object based on a first correction coefficient and an object weight that are preset to correspond to the at least one object. By having at least one processor execute at least one command, the electronic device can correct the luminance of the background based on a second correction coefficient and a background weight that are preset to correspond to the background, thereby obtaining a corrected image.

[0276] In one embodiment of the present disclosure, by having at least one processor execute at least one command, the electronic device can further obtain image weights for correcting the luminance of the entire image through a second artificial intelligence model based on the element of interest and concentration information. By having at least one processor execute at least one command, the electronic device can obtain a corrected image based on the object weights, background weights, and image weights.

[0277] In one embodiment of the present disclosure, the second artificial intelligence model may be a pre-trained artificial intelligence model through reinforcement learning using rewards to infer object weights, background weights, and image weights by receiving interest elements and concentration information as inputs. By having at least one processor execute at least one command, the electronic device may correct the brightness of at least one object based on a first correction coefficient, an object weight, and an image weight that are preset to correspond to at least one object. By having at least one processor execute at least one command, the electronic device may correct the brightness of the background based on a second correction coefficient, a background weight, and an image weight that are preset to correspond to the background, thereby obtaining a corrected image.

[0278] In order to solve the above-described technical problem, one embodiment of the present disclosure provides an operating method of an electronic device. The operating method of the electronic device may include a step of determining at least one element of interest of a user among at least one object and background, based on segmentation information based on at least one object and background included in an image and gaze information acquired through a camera. The operating method of the electronic device may include a step of acquiring concentration information corresponding to at least one element of interest through a first artificial intelligence model, based on at least one user's face image acquired through the camera, brightness information of the image, and illuminance information acquired through an illuminance sensor. The operating method of the electronic device may include a step of acquiring a corrected image in which the brightness of the image is corrected according to a correction weight acquired through a second artificial intelligence model, based on the element of interest and the concentration information.

[0279] In one embodiment of the present disclosure, the camera is an RGB camera, and at least one user's face image may be an RGB image. The method of operating the electronic device may further include a step of converting the face image into a black-and-white image to obtain a converted face image. In the step of obtaining concentration information, concentration information may be obtained through a first artificial intelligence model based on the converted face image, brightness information, and illuminance information.

[0280] In one embodiment of the present disclosure, the first artificial intelligence model may be a pre-trained artificial intelligence model that receives a facial image, brightness information, and illuminance information as input using a training data set and infers concentration information. The training data set may include brightness of a training image, illuminance information of an electronic device providing the training image, a facial image of at least one user viewing the training image, and concentration information obtained from at least one user viewing the training image.

[0281] In one embodiment of the present disclosure, a method of operating an electronic device may include obtaining graph information including a plurality of nodes and a plurality of edges representing relationships between the plurality of nodes, based on segmentation information, interest factors, and concentration information. In the step of obtaining correction weights, the correction weights may be obtained through a second artificial intelligence model based on the graph information.

[0282] In one embodiment of the present disclosure, the plurality of nodes may include a first node corresponding to at least one object, a second node corresponding to a background, and a third node corresponding to at least one user. The plurality of edges may include a first edge from the first node toward the second node, a first edge having a first edge weight, and a second edge from the third node toward either the first node or the second node, and a second edge having a second edge weight.

[0283] In one embodiment of the present disclosure, the second artificial intelligence model may be a pre-trained artificial intelligence model that receives interest elements and concentration information as input and infers object weights and background weights through reinforcement learning using rewards. The reward may include the difference between first concentration information obtained from at least one user viewing a training video and second concentration information obtained from at least one user viewing a training-corrected video in which the luminance of the training video is corrected.

[0284] In one embodiment of the present disclosure, the correction weights may include an object weight corresponding to the luminance correction of at least one object and a background weight corresponding to the luminance correction of the background. In the step of obtaining a correction image, a correction image in which the luminance of the image is corrected based on the object weights and the background weights may be obtained.

[0285] In one embodiment of the present disclosure, the operating method of an electronic device may further include a step of obtaining image weights for correcting the luminance of the entire image through a second artificial intelligence model based on interest element and concentration information. In the step of obtaining the corrected image, the corrected image may be obtained based on object weights, background weights, and image weights.

[0286] In one embodiment of the present disclosure, the second artificial intelligence model may be a pre-trained artificial intelligence model that receives interest element and concentration information as input and uses reinforcement learning with rewards to infer object weights, background weights, and image weights. In the step of obtaining a corrected image, the luminance of at least one object may be corrected based on a first correction coefficient, an object weight, and an image weight that are preset to correspond to at least one object, and the luminance of the background may be corrected based on a second correction coefficient, a background weight, and an image weight that are preset to correspond to the background, thereby obtaining a corrected image.

[0287] In order to solve the above-described technical problem, a computer-readable recording medium having recorded thereon a program for performing at least one method of an embodiment of an operating method of an electronic device disclosed in the present disclosure on a computer can be provided.

[0288] The program executed by the electronic device described in this disclosure may be implemented as hardware components, software components, and / or a combination of hardware components and software components. The program may be executed by any system capable of executing computer-readable instructions.

[0289] Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing device to do a desired thing or may independently or collectively command a processing device to do a desired thing.

[0290] Software may be implemented as a computer program containing instructions stored on a computer-readable storage medium. Examples of computer-readable storage media include magnetic storage media (e.g., read-only memory (ROM), random-access memory (RAM), floppy disks, hard disks, etc.) and optical readable media (e.g., CD-ROMs, DVDs (Digital Versatile Discs)). The computer-readable storage media may be distributed across network-connected computer systems, so that computer-readable code may be stored and executed in a distributed manner. The storage media may be readable by a computer, stored in a memory, and executed by a processor.

[0291] Computer-readable storage media may be provided in the form of non-transitory storage media. Here, the term "non-transitory storage media" simply means a tangible device that does not contain signals (e.g., electromagnetic waves). This term does not distinguish between cases where data is permanently stored in the storage media and cases where data is temporarily stored. For example, a "non-transitory storage medium" may include a buffer in which data is temporarily stored.

[0292] Additionally, programs according to the embodiments disclosed herein may be provided as part of a computer program product. The computer program product may be traded as a commodity between sellers and buyers.

[0293] A computer program product may include a software program and a computer-readable storage medium storing the software program. For example, a computer program product may include a product in the form of a software program (e.g., a downloadable application) distributed electronically by an electronic device manufacturer or through an electronic marketplace (e.g., the Samsung Galaxy Store). For electronic distribution, at least a portion of the software program may be stored on a storage medium or temporarily created. In this case, the storage medium may be a server of the electronic device manufacturer, a server of the electronic marketplace, or a storage medium of an intermediary server that temporarily stores the software program.

[0294] Although the embodiments described above have been described with limited examples and drawings, those skilled in the art will appreciate that various modifications and variations can be made based on the above description. For example, appropriate results can still be achieved even if the described techniques are performed in a different order than described, and / or components such as the described computer system or modules are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents.

Claims

1. Light sensor (150); Camera (140); A memory (120) storing at least one instruction; and At least one processor (130) comprising a processing circuit, The electronic device (100) executes the at least one processor (130) individually or collectively the at least one instruction stored in the memory (120), Based on segmentation information based on at least one object and background included in the image and gaze information acquired through the camera (140), at least one element of interest among the at least one object and the background is determined, Based on at least one face image acquired through the camera (140), brightness information of the image, and illuminance information acquired through the illuminance sensor (150), concentration information corresponding to at least one element of interest is acquired through the first artificial intelligence model (127). An electronic device (100) that controls the brightness of the image to be corrected according to a correction weight obtained through a second artificial intelligence model (128) based on the above-mentioned interest factor and the above-mentioned concentration information.

2. In paragraph 1, The above camera (140) is an RGB camera, and the at least one user's face image is an RGB image, The electronic device (100) executes the at least one instruction individually or collectively by the at least one processor (130), Converting the above facial image into a black and white image to obtain a converted facial image, An electronic device (100) that obtains the concentration information through the first artificial intelligence model (127) based on the converted facial image, the brightness information, and the illuminance information.

3. In either of paragraphs 1 or 2, The above first artificial intelligence model (127) is An artificial intelligence model that is pre-trained to infer the concentration information by using a training data set and receiving the facial image, brightness information, and illuminance information as inputs, The training data set is an electronic device (100) including brightness of a training image, illumination information of an electronic device providing the training image, a face image of at least one user viewing the training image, and concentration information obtained from at least one user viewing the training image.

4. In any one of the clauses 1 to 3, The electronic device (100) executes the at least one instruction individually or collectively by the at least one processor (130), Based on the above segmentation information, the interest element, and the concentration information, graph information including a plurality of nodes and a plurality of edges representing relationships between the plurality of nodes is obtained, An electronic device (100) that obtains the correction weight through the second artificial intelligence model (128) based on the above graph information.

5. In paragraph 4, The plurality of nodes include a first node corresponding to the at least one object, a second node corresponding to the background, and a third node corresponding to the at least one user, An electronic device (100) wherein the plurality of edges include a first edge having a first edge weight and a second edge having a second edge weight and a second edge having a second edge weight and a third edge having a second edge weight.

6. In any one of paragraphs 1 to 5, The above second artificial intelligence model (128) is an artificial intelligence model that has been pre-learned through reinforcement learning using rewards to infer the correction weight by receiving the interest factor and the concentration information as inputs. An electronic device (100) wherein the above compensation includes a difference between first concentration information obtained from at least one user viewing a training video and second concentration information obtained from at least one user viewing a training correction video in which the brightness of the training video is corrected.

7. In any one of paragraphs 1 to 6, The above correction weights include an object weight corresponding to the luminance correction of the at least one object and a background weight corresponding to the luminance correction of the background, The electronic device (100) executes the at least one instruction individually or collectively by the at least one processor (130), An electronic device (100) that obtains the corrected image by correcting the brightness of the image based on the object weight and the background weight.

8. In paragraph 7, The electronic device (100) executes the at least one instruction individually or collectively by the at least one processor (130), An electronic device (100) that corrects the brightness of at least one object based on a first correction coefficient preset to correspond to at least one object and the object weight, and corrects the brightness of the background based on a second correction coefficient preset to correspond to the background and the background weight to obtain the corrected image.

9. In either of paragraphs 7 or 8, The electronic device (100) executes the at least one instruction individually or collectively by the at least one processor (130), Based on the above interest elements and the concentration information, an image weight for correcting the brightness of the entire image is further obtained through the second artificial intelligence model (128). An electronic device (100) that obtains the corrected image based on the object weight, the background weight, and the image weight.

10. In paragraph 9, The second artificial intelligence model (128) is an artificial intelligence model that has been pre-learned through reinforcement learning using the reward to infer the object weight, the background weight, and the image weight by receiving the interest element and the concentration information as input. The electronic device (100) executes the at least one instruction individually or collectively by the at least one processor (130), An electronic device (100) that corrects the brightness of the at least one object based on a first correction coefficient preset to correspond to the at least one object, the object weight, and the image weight, and corrects the brightness of the background based on a second correction coefficient preset to correspond to the background, the background weight, and the image weight to obtain the corrected image.

11. In the operating method of an electronic device (100), A step (S300) of determining at least one element of interest of a user among at least one object and the background included in the image based on segmentation information and gaze information obtained through a camera (140); A step (S600) of obtaining concentration information corresponding to at least one element of interest through a first artificial intelligence model (127) based on at least one user's face image obtained through the camera (140), brightness information of the image, and illuminance information obtained through an illuminance sensor (150); and An operating method of an electronic device (100), including a step (S800) of obtaining a corrected image in which the brightness of the image is corrected according to a correction weight obtained through a second artificial intelligence model (128) based on the above-mentioned interest element and the above-mentioned concentration information.

12. In paragraph 11, The above camera (140) is an RGB camera, and the at least one user's face image is an RGB image, The operating method of the above electronic device (100) is: Further comprising a step of converting the above facial image into a black and white image to obtain a converted facial image, In the step (S600) of obtaining the above concentration information, An operating method of an electronic device (100) for obtaining the concentration information through the first artificial intelligence model (127) based on the above-described converted face image, the brightness information, and the illuminance information.

13. In either of paragraphs 11 or 12, The above first artificial intelligence model (127) is An artificial intelligence model that is pre-trained to infer the concentration information by using a training data set and receiving the facial image, brightness information, and illuminance information as inputs, The above training data set is an operating method of an electronic device (100) including the brightness of a training image, illumination information of an electronic device providing the training image, a face image of at least one user viewing the training image, and concentration information obtained from at least one user viewing the training image.

14. In any one of paragraphs 11 to 13, The operating method of the above electronic device (100) is: Further comprising a step of obtaining graph information including a plurality of nodes and a plurality of edges representing relationships between the plurality of nodes based on the segmentation information, the interest element, and the concentration information. In the step of obtaining the above correction weight, An operating method of an electronic device (100) for obtaining the correction weight through the second artificial intelligence model (128) based on the above graph information.

15. A computer-readable recording medium having recorded thereon a program for performing the method of operation described in any one of Articles 11 to 14 on a computer.

Images