Method for predicting user emotion and electronic equipment

By receiving and analyzing the combined representation of users, devices and environmental situations, predicting user emotions and dynamically adjusting the response of electronic devices, solving the problems of inaccurate and insufficient personalization of emotions in the prior art, and achieving more personalized and dynamic user experience.

CN119072688APending Publication Date: 2024-12-03SAMSUNG ELECTRONICS CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202380038038.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-05-04
Filing Date
2023-05-04
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

The prior art is difficult to accurately predict the emotional state of users, which leads to electronic devices being unable to provide personalized and dynamic user experience, neglecting the dynamic factors of the user environment, resulting in a lack of personalization and real-time responses to chatbots and virtual environments.

Method used

By receiving a combination representation of user situations, device situations, and environmental situations, multiple models are used to predict the user's emotional state and adjust the user experience, personalized content and virtual environment parameters based on the predicted emotions.

Benefits of technology

It realizes accurate prediction and personalized response to user emotions, enhances the user experience, and provides a more dynamic and personalized user interaction and virtual environment.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119072688A_ABST
    Figure CN119072688A_ABST
Patent Text Reader

Abstract

A method for predicting an emotion of a user by an electronic device is provided. The method includes receiving, by an electronic device, a user context, a device context, and an environmental context from the electronic device and one or more other electronic devices connected to the electronic device, and determining, by the electronic device, a combined representation of the user context, the device context, and the environmental context. The method further includes determining, by the electronic device, a plurality of user characteristics based on the combined representation of the user context, the device context, and the environmental context; and predicting, by the electronic device, an emotion of the user based on the combined representation of the user context, the device context, the environmental context, and the plurality of user characteristics.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to electronic devices, and more particularly to methods and electronic devices for predicting a user's emotions.This application is based on and claims priority from Indian Provisional Application No. 202241026033 filed on May 4, 2022, the disclosure of which is incorporated herein by reference. Background Art

[0002] As technology advances, electronic devices have become an obvious part of human life. Users of electronic devices are usually static, but the emotional state of the user can be dynamic. However, the emotional state of the user needs to be considered to provide a better user experience to the user of the electronic device.

[0003] Various existing methods for user activities (such as, for example, conversations between chatbots and users) are mechanical in nature with low personalization. Existing methods determine the user's emotional state based on modalities such as, for example, text, voice, images, etc.; user actions such as, for example, avatar personification and user ratings (UI), which may not provide the user's accurate emotional state.

[0004] In virtual environments and applications, user personality and emotional state are not properly reflected in real time, which fails to provide a comprehensive representation of the user. Various existing methods for determining a user's emotional state do not take into account dynamic factors such as the user's environment, and thus conversational applications such as chatbots, avatars, etc. provide similar responses regardless of the user's emotional state. Therefore, it is desirable to at least provide a mechanism that does not have the above-mentioned problems. Summary of the invention

[0005] Technical issues

[0006] A method and electronic device for predicting a user's emotions through an electronic device are provided. The proposed method predicts the user's emotions based on various dynamic parameters, including but not limited to data associated with the electronic device and the user's life pattern using various models across multiple devices of the user. Therefore, unlike traditional methods and systems, in the proposed method, the predicted emotions can be used to dynamically change the options provided to the user based on the user's emotions. Therefore, the proposed method enhances the user experience and provides personalization and various functions of the electronic device based on the user's emotions.

[0007] Technical Solutions

[0008] A method for predicting a user's emotions through an electronic device is provided. According to one aspect of the present disclosure, a method for predicting a user's emotions through an electronic device includes: receiving a user context, a device context, and an environmental context through an electronic device, wherein the user context, the device context, and the environmental context are collected through at least one of the electronic device and at least one of one or more other electronic devices connected to the electronic device; determining, by the electronic device, a combined representation of the user context, the device context, and the environmental context; determining, by the electronic device, a plurality of user characteristics based on the combined representation of the user context, the device context, and the environmental context; and predicting the user's emotions through the electronic device based on the plurality of user characteristics and the combined representation of the user context, the device context, and the environmental context.

[0009] The method for predicting a user's emotions via an electronic device may also include: performing, via the electronic device, at least one of the following based on the predicted user's emotions: modifying the user experience on the electronic device and on at least one of one or more other electronic devices, personalizing content on the electronic device and on at least one of one or more other electronic devices, generating at least one object for providing emotional support to the user using an emotion profile on the electronic device and on at least one of one or more other electronic devices, providing safety functions to the user in a virtual environment, and modifying at least one user parameter in the virtual environment.

[0010] The method for predicting a user's emotions via an electronic device may also include: determining via the electronic device at least one of: the user's consumption of content, abnormal usage patterns on the electronic device or on at least one of one or more other electronic devices, recurring activities performed by the user on the electronic device or on at least one of one or more other electronic devices, and duration spent by the user on the electronic device or on at least one of the one or more other electronic devices; and determining via the electronic device a quality of the predicted user emotion, wherein the quality of the predicted emotion is a positive emotion or a negative emotion.

[0011] Determining multiple user characteristics based on a combined representation of user context, device context, and environmental context by the electronic device may include: providing the combined representation of user context, device context, and environmental context to a first network and multiple intermediate models by the electronic device; and determining multiple user characteristics by the electronic device.

[0012] The method for predicting the emotions of a user by an electronic device may further include predicting, by the electronic device, a first intermediate emotion set based on a plurality of user characteristics and a combined representation of a user context, a device context, and an environmental context.

[0013] The method for predicting a user's emotions through an electronic device may also include: providing a combined representation of user context, device context and environmental context to a second network and a third network through the electronic device; determining, by the electronic device, a local map emotion prediction from the second network and a global node prediction from the third network; combining, by the electronic device, the local map emotion prediction and the global node prediction based on specific weights; and predicting a second intermediate emotion set through the electronic device.

[0014] Determining a combined representation of user context, device context, and environmental context by an electronic device may include: determining, by the electronic device, a plurality of features associated with the user based on the user context, the device context, and the environmental context; dividing, by the electronic device, the plurality of features associated with the user into a plurality of categories corresponding to specific time durations; generating, by the electronic device, at least one vector representation for each of the plurality of categories using encoding; and determining, by the electronic device, a combined representation of user context, device context, and environmental context based on at least one vector representation for each of the plurality of categories.

[0015] Predicting a user's emotions based on a combined representation of user context, device context, environmental context, and multiple user characteristics by an electronic device may include: receiving a first intermediate emotion set and a second intermediate emotion set by at least one second model of the electronic device; receiving a classification clustering map by the at least one second model of the electronic device; performing an integration technique on the first intermediate emotion set and the second intermediate emotion set based on the classification clustering map by the at least one second model of the electronic device; and predicting the user's emotions by the electronic device.

[0016] At least one first model may be used to determine a plurality of user characteristics, and at least one second model may be used to predict a user's emotion.

[0017] According to one aspect of the present disclosure, a method for predicting a user's emotions through an electronic device includes: receiving first data through the electronic device, the first data including user activities, operating status of the electronic device, and operating status of at least one of one or more other electronic devices connected to the electronic device; receiving second data representing demographic characteristics and lifestyle of the user by the electronic device, wherein the second data is collected from at least one of the electronic device and at least one of the one or more other electronic devices connected to the electronic device; normalizing the first data and the second data by the electronic device for input into multiple models; predicting multiple user characteristics from the models by the electronic device; and predicting the user's emotions through the electronic device based on the first data, the second data, and the multiple user characteristics.

[0018] According to one aspect of the present disclosure, an electronic device for predicting user emotions includes: at least one memory configured to store at least one instruction; at least one processor communicating with the at least one memory; and a communicator communicating with the at least one memory and the at least one processor, wherein the at least one processor is configured to execute at least one instruction to: receive user context, device context, and environmental context, wherein the user context, device context, and environmental context are collected by at least one of the electronic device and one or more other electronic devices connected to the electronic device; determine a combined representation of the user context, the device context, and the environmental context; determine multiple user characteristics based on the combined representation of the user context, the device context, and the environmental context; and predict the user's emotions based on the multiple user characteristics and the combined representation of the user context, the device context, and the environmental context.

[0019] At least one processor of the electronic device may also be configured to execute at least one instruction to: perform at least one of the following based on the predicted emotion of the user: modify the user experience on the electronic device and at least one of one or more other electronic devices, personalize the content on the electronic device and at least one of one or more other electronic devices, utilize the emotion profile on the electronic device and at least one of one or more other electronic devices to generate at least one object for providing emotional support to the user, provide safety functions to the user in a virtual environment; and modify at least one user parameter in the virtual environment.

[0020] At least one processor of the electronic device may also be configured to execute at least one instruction to: determine at least one of: a user's consumption of content, an unusual usage pattern on the electronic device or at least one of the one or more other electronic devices, a recurring activity performed by the user on the electronic device or at least one of the one or more other electronic devices, and a duration of time spent by the user on the electronic device or at least one of the one or more other electronic devices; and determine a quality of a predicted user emotion, wherein the quality of the predicted emotion is a positive emotion or a negative emotion.

[0021] At least one processor of the electronic device may also be configured to execute at least one instruction to: determine a plurality of user characteristics based on the combined representation of the user context, the device context, and the environmental context by providing the combined representation of the user context, the device context, and the environmental context to the first network and the plurality of intermediate models.

[0022] At least one processor of the electronic device may also be configured to execute at least one instruction to: determine a combined representation of the user context, the device context, and the environmental context by: determining a plurality of features associated with the user from the user context, the device context, and the environmental context, separating the plurality of features associated with the user into a plurality of categories corresponding to specific time durations, generating at least one vector representation for each of the plurality of categories, and determining the combined representation of the user context, the device context, and the environmental context based on the at least one vector representation for each of the plurality of categories.

[0023] The electronic device according to claim 15, wherein the at least one processor is further configured to execute the at least one instruction to: predict the user's emotions based on the combined representation of the multiple user characteristics and the user context, the device context, and the environmental context in the following manner: receiving a first intermediate emotion set and a second intermediate emotion set by at least one second model of the electronic device; receiving a classification clustering map by the at least one second model of the electronic device; performing an integration technique on the first intermediate emotion set and the second intermediate emotion set based on the classification clustering map by the at least one second model of the electronic device; and predicting the user's emotions by the electronic device.

[0024] At least one processor of the electronic device may also be configured to execute at least one instruction to: determine multiple user characteristics based on a combined representation of user context, device context, and environmental context using at least one first model, and predict the user's emotions using at least one second model.

[0025] The at least one processor of the electronic device may also be configured to execute at least one instruction to: predict a first intermediate emotion set based on a plurality of user characteristics and a combined representation of user context, device context, and environmental context.

[0026] At least one processor of the electronic device may also be configured to execute at least one instruction to: provide a combined representation of user context, device context, and environmental context to a second network and a third network; determine a local graph emotion prediction from the second network and a global node prediction from the third network; combine the local graph emotion prediction and the global node prediction based on specific weights; and predict a second intermediate emotion set.

[0027] According to one aspect of the present disclosure, an electronic device for predicting a user's emotions includes: at least one memory configured to store at least one instruction; at least one processor communicating with the at least one memory; and a communicator communicating with the at least one memory and the at least one processor, wherein the at least one processor is configured to execute the at least one instruction to: receive first data, the first data including user activities, the operating state of the electronic device, and the operating state of at least one of one or more other electronic devices connected to the electronic device; receive second data representing demographic characteristics and lifestyle of the user, wherein the second data is collected from at least one of the electronic device and at least one of the one or more other electronic devices connected to the electronic device; normalize the first data and the second data for input into multiple models; predict multiple user characteristics from the models; and predict the user's emotions based on the first data, the second data, and the multiple user characteristics.

[0028] These and other aspects of the embodiments disclosed herein will be better appreciated and understood when considered in conjunction with the following description and accompanying drawings. However, it should be understood that the following description, although indicating preferred embodiments and many of their specific details, is given by way of illustration and not limitation. Many changes and modifications may be made within the scope of the embodiments herein, and the embodiments herein include all such modifications. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] These and other features, aspects and advantages of the present disclosure are illustrated in the accompanying drawings, throughout which the same reference numerals indicate corresponding parts in the various drawings. The embodiments of the present invention will be better understood from the following description with reference to the accompanying drawings, in which:

[0030] FIG1 is a block diagram of an electronic device for predicting a user's emotions according to an embodiment of the present invention;

[0031] 2 is a flow chart showing a method for predicting a user's emotions through an electronic device according to an embodiment of this document;

[0032] 3 is another flow chart showing a method for predicting a user's emotions by an electronic device according to an embodiment of the present invention;

[0033] FIG. 4 is an example of an inference process for predicting a user's emotions through an electronic device according to an embodiment of the present invention;

[0034] 5A-5B illustrate various actions involved in predicting emotions through an electronic device according to an embodiment of the present invention;

[0035] 6A-6C are examples showing aggregation of device context, environment context, and user context by a context aggregation manager according to an embodiment of this document;

[0036] 7 is a block diagram illustrating classification data interpretation and presentation by a sentiment prediction manager according to embodiments herein;

[0037] FIG8 is a block diagram illustrating a classification fusion network (CaFuNet) of a sentiment prediction manager according to embodiments herein;

[0038] 9 is a block diagram illustrating a CaFuNet with a user comprehension model of a sentiment prediction manager according to an embodiment of the present invention;

[0039] FIG10 is a flow chart illustrating the functionality of a deep neural network (DNN) like an autoencoder network to capture semantic information according to an embodiment of the present invention;

[0040] FIG11 is a block diagram illustrating an autoencoder network according to an embodiment of the present invention;

[0041] FIG. 12 is a block diagram illustrating the functionality of a DNN like a graph isomorphic network (GIN) with a graph convolutional network (GCN) to capture structural and sequential information according to an embodiment of the present invention;

[0042] FIG13 is a flow chart illustrating a graph operation according to an embodiment of the present invention;

[0043] FIG14 shows an example of an insight level experience provided based on predicted user emotions according to an embodiment of the present invention;

[0044] 15A-15D illustrate various examples of personalization of an electronic device based on predicted user emotions according to embodiments herein;

[0045] 16A to 16D illustrate examples of creating a digital twin in a virtual environment using predicted emotions according to embodiments herein;

[0046] FIG. 17A illustrates an example environment modeling scenario according to embodiments herein; and

[0047] FIG. 17B illustrates another example environment modeling scenario according to embodiments herein. DETAILED DESCRIPTION

[0048] The embodiments of this article and its various features and advantageous details are explained more fully with reference to the non-limiting embodiments shown in the drawings and described in detail in the following description. The description of known components and processing techniques is omitted so as not to unnecessarily obscure the embodiments of this article. In addition, the various embodiments described herein are not necessarily mutually exclusive, because some embodiments can be combined with one or more other embodiments to form new embodiments. Unless otherwise stated, the term "or" as used herein refers to a non-exclusive or. The examples used herein are intended only to facilitate understanding of the manner in which the embodiments of this article can be practiced, and further enable those skilled in the art to practice the embodiments of this article. Therefore, the examples should not be interpreted as limiting the scope of the embodiments of this article.

[0049] As conventional in the art, embodiments may be described and illustrated according to blocks that perform one or more functions described. These blocks (which may be referred to herein as units or modules, etc.) are physically implemented by analog or digital circuits (such as logic gates, integrated circuits, microprocessors, microcontrollers, memory circuits, passive electronic components, active electronic components, optical components, hard-wired circuits, etc.), and may optionally be driven by firmware. The circuit may be, for example, embodied in one or more semiconductor chips, or embodied on a substrate support such as a printed circuit board. The circuits constituting the blocks may be implemented by dedicated hardware, or by a processor (e.g., one or more programmed microprocessors and associated circuits), or by a combination of dedicated hardware that performs some functions of the blocks and processors that perform other functions of the blocks. Without departing from the scope of the present disclosure, each block of the embodiment may be physically divided into two or more interactive and discrete blocks. Similarly, without departing from the scope of the present disclosure, the blocks of the embodiment may be physically combined into more complex blocks.

[0050] The accompanying drawings are used to help easily understand various technical features, and it should be understood that the embodiments presented herein are not limited by the accompanying drawings. Therefore, the present disclosure should be interpreted as extending to any changes, equivalents and substitutes other than those specifically set forth in the accompanying drawings. Although the terms "first", "second" etc. can be used herein to describe various elements, these elements should not be limited by these terms. These terms are usually only used to distinguish one element from another element.

[0051] It should be understood that when an element is referred to as being “connected” or “connected to” another element, it may be directly or indirectly connected to the other element, where the indirect connection includes “connection via a wireless communication network”.

[0052] Herein, the expression "at least one of a, b, or c" means only a, only b, only c, both a and b, both a and c, both b and c, or all of a, b, and c.

[0053] Referring now to the drawings, and more particularly to FIGS. 1 to 17B , where like reference numerals denote corresponding features throughout the drawings, FIG. 1 is a block diagram of an electronic device 100 for predicting a user's emotions according to an embodiment of the present invention. Referring to FIG. 1 , the electronic device 100 may be, for example, but not limited to, a smart phone, a mobile phone, a tablet computer, a laptop computer, a handheld computer, an AI speaker, an IoT sensor, an intelligent social robot, a personal digital assistant (PDA), a music player, a video player, a wearable device, etc.

[0054] In an embodiment, the electronic device 100 includes a memory 120, a processor 140, a communicator 160, an emotion management controller 180, and a display 190. The emotion management controller 180 is implemented by processing circuits such as logic gates, integrated circuits, microprocessors, microcontrollers, memory circuits, passive electronic components, active electronic components, optical components, hard-wired circuits, etc., and may be optionally driven by firmware. The circuits may be embodied, for example, in one or more semiconductors.

[0055] The memory 120 is configured to store instructions to be executed by the processor 140. The memory 120 may include a non-volatile storage element. Examples of such non-volatile storage elements may include a magnetic hard disk, an optical disk, a floppy disk, a flash memory, or an electrically programmable memory (EPROM) or an electrically erasable programmable (EEPROM) memory. In addition, in some examples, the memory 120 may be considered a non-transitory storage medium. The term "non-transitory" may indicate that the storage medium is not embodied in a carrier or propagation signal. However, the term "non-transitory" should not be interpreted as the memory 120 being non-removable. In some examples, the memory 120 may be configured to store a larger amount of information. In some instances, a non-transitory storage medium may store data that may change over time (e.g., in a random access memory (RAM) or a cache memory).

[0056] The processor 140 communicates with the memory 120, the communicator 160, and the emotion management controller 180. The processor 140 is configured to execute instructions stored in the memory 120 and perform various processes. The processor may include one or more processors, which may be a general-purpose processor (such as a central processing unit (CPU), an application processor (AP), etc.), a graphics processing unit (such as a graphics processing unit (GPU)), a visual processing unit (VPU), and / or an artificial intelligence (AI) dedicated processor (such as a neural processing unit (NPU)).

[0057] The communicator 160 includes electronic circuits specific to a standard for implementing wired or wireless communication. The communicator 160 is configured to communicate internally between internal hardware components of the electronic device 100 and to communicate with external devices via one or more networks.

[0058] In an embodiment, the emotion management controller 180 includes a context aggregation manager 182 , an emotion prediction manager 184 , and an emotion-based content manager 186 .

[0059] The context aggregation manager 182 is configured to receive user context, device context, and environmental context from the electronic device 100 and one or more other electronic devices 100 connected to the electronic device 100, and determine an aggregated version of each of the received user context, device context, and environmental context.

[0060] The emotion prediction manager 184 is configured to provide a combined representation of a user context, a device context, and an environmental context to the first network 184a and a plurality of intermediate models, and to determine a plurality of user characteristics based on the combined representation of the user context, the device context, and the environmental context. Determining the combined representation of the user context, the device context, and the environmental context includes determining a plurality of features associated with the user from the user context, the device context, and the environmental context, and separating the plurality of features associated with the user into a plurality of categories within a specific duration. In addition, the emotion prediction manager 184 is configured to generate at least one vector representation for each of the plurality of categories using the encoding, and to determine the combined representation of the user context, the device context, and the environmental context based on at least one vector representation of each of the plurality of categories. The plurality of user characteristics are determined using at least one first model, and wherein at least one second model is used to predict the user's emotions.

[0061] In addition, the emotion prediction manager 184 is configured to provide the combined representation of the user context, the device context, and the environmental context to the second network 184b and the third network 184c, and determine the local graph emotion prediction from the second network 184b and the global node prediction from the third network 184c. The third network 184c is, for example, a graph convolutional network (GCN). The emotion prediction manager 184 is configured to combine the local graph emotion prediction and the global node prediction based on a specific weight, and predict a second intermediate emotion set.

[0062] In addition, the emotion prediction manager 184 is configured to provide the first intermediate emotion set and the second intermediate emotion set and the classification cluster map to at least one second model. The emotion prediction manager 184 predicts the first intermediate emotion set based on a combination of multiple user characteristics and user context, device context, and environmental context. In addition, the emotion prediction manager 184 is configured to perform an integration technique on the first intermediate emotion set and the second intermediate emotion set based on the classification cluster map and predict the user's emotion.

[0063] In addition, the emotion prediction manager 184 is configured to determine at least one of the following: the user's consumption of content, unusual usage patterns on the electronic device 100 or one or more other electronic devices 100, recurring activities performed by the user on the electronic device 100 or one or more other electronic devices 100, and the duration of time the user spends on the electronic device 100 or one or more other electronic devices 100, and determine the quality of the emotion of the particular feature. The quality of the predicted emotion is a positive emotion or a negative emotion.

[0064] In another embodiment, the emotion prediction manager 184 is configured to receive first data including user activities and operation states of the electronic device 100 and one or more other electronic devices 100a-N connected to the electronic device 100, and receive second data representing demographic characteristics and lifestyles of the users from the electronic device 100 and one or more other electronic devices 100a-N connected to the electronic device 100. The emotion prediction manager 184 is configured to normalize the first data and the second data to feed into a plurality of models; predict a plurality of user characteristics from the models; and predict the user's emotion based on the first data, the second data, and the plurality of user characteristics.

[0065] The emotion-based content manager 186 is configured to modify the user experience on the electronic device 100 and one or more other electronic devices 100a-N or personalize the content on the electronic device 100 and one or more other electronic devices 100a-N based on the predicted user emotions. The emotion-based content manager 186 may also be configured to perform emotion profiling on the electronic device 100 and one or more other electronic devices 100a-N, or generate at least one object for providing emotional support to the user, or provide safety to the user in the virtual environment, or modify at least one user parameter in the virtual environment.

[0066] Personalization of content on the electronic device 100 includes providing animation applications based on predicted user emotions, such as keyboard-based applications. For example, fast keyboard animations for negative emotions (anxiety, sadness) and smooth animations for positive emotions (happiness, excitement). In addition, personalization of content on the electronic device 100 includes providing dynamic emotion-based locks. For example, when the predicted user emotion is anxiety, the screen lock duration is increased because the user tends to check the smartphone frequently. Another example includes customization of overlay screens based on predicted user emotions. Personalization of content on the electronic device 100 includes wallpaper selection based on predicted user emotions. For example, happy images across albums are provided on the electronic device 100.

[0067] Another example of personalization of content on the electronic device 100 includes automatic device color palette personalization based on predicted user emotions. For example, when the predicted emotion for the user is anger, the device color palette can become red, yellow can be used for happiness, black can be used for fear, etc.

[0068] In another example, personalization of content on the electronic device 100 is provided by prioritizing between emotional interactions and performance-based interactions based on predicted user emotions and his / her response to personalization. For example, when the user is happy and excited, then emotional appeal outweighs performance. Therefore, the electronic device 100 provides various animations, vivid color palette themes, etc.

[0069] When the user is in a hurry or nervous, performance requirements outweigh emotional appeal. Therefore, the electronic device 100 does not provide animation, displays a single theme such as dark mode, etc., so that performance is higher.

[0070] The emotion-based content manager 186 provides an insight-level experience. For example, the emotion-based content manager 186 provides insights to the user, such as, for example, who the user is happy with based on a conversation or call, which application usage makes the user very happy (or sad), and then based on the insight-level experience, the user can choose to install or uninstall the application accordingly.

[0071] At least one of the multiple modules / components of the emotion management controller 180 may be implemented by an AI model. Functions associated with the AI ​​model may be performed by the memory 120 and the processor 140. One or more processors control the processing of input data according to predefined operating rules or AI models stored in non-volatile memory and volatile memory. Predefined operating rules or artificial intelligence models are provided through training or learning.

[0072] Here, providing by learning means forming a predefined operating rule or AI model of desired characteristics by applying a learning process to a plurality of learning data. The learning can be performed in the device itself that executes the AI ​​according to the embodiment, and / or can be implemented by a separate server / system.

[0073] An AI model can be composed of multiple neural network layers. Each layer has multiple weight values, and the layer operation is performed by the calculation of the previous layer and the operation of multiple weights. Examples of neural networks include, but are not limited to, convolutional neural networks (CNNs), deep neural networks (DNNs), recurrent neural networks (RNNs), restricted Boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), generative adversarial networks (GANs), and deep Q networks.

[0074] The learning process is a method for training a predetermined target device (e.g., a robot) using a plurality of learning data to enable, allow, or control the target device to make a determination or prediction. Examples of the learning process include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0075] In an embodiment, the display 190 is configured to display personalized content based on the predicted user emotion of the electronic device 100. The display 190 can receive input and is made of one of a liquid crystal display (LCD), a light emitting diode (LED), an organic light emitting diode (OLED), etc.

[0076] Although FIG. 1 shows various hardware components of the electronic device 100, it should be understood that other embodiments are not limited thereto. In other embodiments, the electronic device 100 may include a greater or lesser number of components. In addition, the markings or names of the components are for illustrative purposes only and do not limit the scope of the present disclosure. One or more components may be combined to perform the same or substantially similar function as predicting the user's emotions through the electronic device 100.

[0077] FIG. 2 is a flowchart illustrating a method 200 for predicting a user's emotion through the electronic device 100 according to an embodiment of this document.

[0078] 2 , at operation 202, the method 200 includes the electronic device 100 receiving a user context, a device context, and an environmental context from the electronic device 100 and one or more other electronic devices 100a-N connected to the electronic device 100. For example, in the electronic device 100 described in FIG. 1 , the emotion management controller 180 is configured to receive a user context, a device context, and an environmental context from the electronic device 100 and one or more other electronic devices 100a-N connected to the electronic device 100.

[0079] At operation 204, the method 200 includes the electronic device 100 determining a combined representation of the user context, the device context, and the environmental context. For example, in the electronic device 100 depicted in FIG. 1 , the emotion management controller 180 is configured to determine a combined representation of the user context, the device context, and the environmental context.

[0080] At operation 206, the method 200 includes the electronic device 100 determining a plurality of user characteristics based on a combined representation of the user context, the device context, and the environmental context. For example, in the electronic device 100 depicted in FIG. 1 , the emotion management controller 180 is configured to determine a plurality of user characteristics based on a combined representation of the user context, the device context, and the environmental context.

[0081] At operation 208, the method includes the electronic device 100 predicting the user's emotions based on the user context, the device context, the combined representation of the environmental context, and the plurality of user characteristics. For example, in the electronic device 100 depicted in FIG. 1 , the emotion management controller 180 is configured to predict the user's emotions based on the user context, the device context, the combined representation of the environmental context, and the plurality of user characteristics.

[0082] The various actions, behaviors, blocks, operations, etc. in the flowchart of Figure 2 can be performed in the order presented, in a different order, or simultaneously. In addition, in some embodiments, some of the actions, behaviors, blocks, operations, etc. can be omitted, added, modified, skipped, etc. without departing from the scope of the present disclosure.

[0083] FIG. 3 is another flow chart illustrating a method 300 for predicting a user's emotion through the electronic device 100 according to an embodiment of this document.

[0084] 3 , at operation 302 , the method includes the electronic device 100 receiving first data including user activities and operating states of the electronic device 100 and one or more other electronic devices 100 a-N connected to the electronic device 100. The user activities may be, for example, user responses in a specific application, a user ordering a specific kind of food at a specific time, a user going to the gym, a user routine, etc. The operating state of the electronic device 100 may be, for example, but not limited to, connection capability to Wi-Fi or Bluetooth, battery capability, available storage, video call availability or unavailability, etc. For example, in the electronic device 100 described in FIG. 1 , the emotion management controller 180 is configured to receive first data including user activities and operating states of the electronic device 100 and one or more other electronic devices 100 a-N connected to the electronic device 100.

[0085] At operation 304, the method includes the electronic device 100 receiving second data representing demographic characteristics and lifestyles of the user from the electronic device 100 and one or more other electronic devices 100a-N connected to the electronic device 100. For example, in the electronic device 100 described in FIG. 1 , the emotion management controller 180 is configured to receive second data representing demographic characteristics and lifestyles of the user from the electronic device 100 and one or more other electronic devices 100a-N connected to the electronic device 100.

[0086] At operation 306, the method 300 includes the electronic device 100 normalizing the first data and the second data to feed to multiple models. For example, in the electronic device 100 described in FIG1, the emotion management controller 180 is configured to normalize the first data and the second data to feed to multiple models.

[0087] At operation 308, the method 300 includes the electronic device 100 predicting a plurality of user characteristics from the model. For example, in the electronic device 100 depicted in FIG1, the emotion management controller 180 is configured to predict a plurality of user characteristics according to the model.

[0088] At operation 310, the method 300 includes the electronic device 100 predicting the user's emotions based on the first data, the second data, and a plurality of user characteristics. For example, in the electronic device 100 described in FIG. 1, the emotion management controller 180 is configured to predict the user's emotions based on the first data, the second data, and a plurality of user characteristics.

[0089] The various actions, behaviors, blocks, operations, etc. in the flowchart of Figure 3 can be performed in the order presented, in a different order, or simultaneously. In addition, in some embodiments, some of the actions, behaviors, blocks, operations, etc. can be omitted, added, modified, skipped, etc. without departing from the scope of the present disclosure.

[0090] FIG. 4 is a flowchart showing an example of an inference flow for predicting a user's emotion by the electronic device 100 according to an embodiment described herein.

[0091] Referring to FIG. 4 , various users and various data availabilities are considered. At operation 401 , consider the case of user A, where first data (e.g., data associated with the electronic device 100 ) is available together with second data as demographic data. A first model is used to infer the personality of the user. The user personality data is then provided to the emotion prediction manager 184 together with the first data and the second data to predict the emotion of user A.

[0092] Similarly, at operation 402, consider the case of user B, where first data (e.g., data associated with the electronic device 100) is available together with second data, which is demographic data without gender details of user B. The first model is used to infer the personality of the user and the gender details of user B. The user personality data and the gender details of user B are then provided to the emotion prediction manager 184 along with the first data and the second data to predict the emotion of user B.

[0093] Similarly, at operation 403, consider the case of user C, where first data (e.g., data associated with the electronic device 100) is available together with second data, which is demographic data without age details of user C. The first model is used to infer the personality of the user along with the age details of user C. The user personality data and the age details of user C are then provided to the emotion prediction manager 184 along with the first data and the second data to predict the emotion of user C.

[0094] 5A-5B illustrate various actions involved in predicting emotions by the electronic device 100 according to embodiments described herein.

[0095] Referring to FIG. 5A , at operation 501a, first data and second data are provided to the electronic device 100. At operation 502a, classification features are determined, and at operation 503a, classification features are separated by each instance / time window. At operation 504a, encoding and vector generation are performed. At operation 505a, a combined representation is generated, and at operation 506a, normalized data is generated. At operation 507a, the normalized data passes through a highway network, and at operation 508a, encoder embedding is performed. In addition, the encoded data is sent through a first network 184a and a second network 184b, and the first network 184a can be, for example, an autoencoder network 184a, and the second network 184b can be, for example, a graph isomorphism network 184b. At operation 510a, outputs from both the first network 184a and the second network are sent through a fully connected layer, and at operation 511a, the user's emotions and emotional quality are predicted.

[0096] Referring to FIG. 5B , in conjunction with FIG. 5A , operations 501a-507a are the same as 501b-507b. However, FIG. 5B includes a demographic feature prediction ML model (age / gender) and a personality prediction ML model. Here, the demographic feature prediction ML model and the personality prediction ML model can be used independently to determine demographic feature data or personality-related data. This pipeline is used when the first data and the second data are missing some user information (such as the user's gender, the user's age, etc.).

[0097] 6A-6C are examples showing aggregation of device context, environmental context, and user context by the context aggregation manager 182 according to embodiments described herein.

[0098] Referring to FIG. 6A , consider a user's various electronic devices 100a-N, such as a smartphone 100a, a wearable device 100b, a robot 100c, and other IoT devices 100d-N. At operation 601, the context aggregation manager 182 receives a device context from the smartphone 100a. Here, the device context is determined based on, for example, but not limited to, the use of Wi-Fi, GPS, microphone, etc. At operation 602, the context aggregation manager 182 receives a device context from the wearable device 100b, which includes the heart rate, blood pressure, sleep pattern, etc. associated with the user. At operation 603, the context aggregation manager 182 receives a device context from the robot program 100c, which includes details such as available videos, images, audio files, etc. Similarly, at operation 604, the context aggregation manager 182 receives a device context from the user's other IoT devices 100d. At operation 605, the data preprocessor 182a performs data filtering and arrangement. At operation 606, the context aggregator 182b aggregates similar data received from different devices of the user.

[0099] Similarly, in FIG. 6B , environmental context is received and aggregated, which includes various environmental factors, such as, but not limited to, location, time, weather conditions, temperature, humidity, etc.

[0100] Similarly, in FIG. 6C , user context is received and aggregated, including economic and social factors (such as culture, finance, assets, etc.) as well as historical factors of the user (such as, for example, personality, situational emotions, family, decision-making, etc.).

[0101] FIG. 7 is a block diagram illustrating the interpretation and representation of categorical data by the emotion prediction manager 184 according to an embodiment described herein. Referring to FIG. 7 , at operation 702, the first data and the second data are separated into categorical features, continuous features, discrete features, and derived (or secondary) features. At operation 704, feature separation for each instance / time window is performed. At operation 706, encoding and vector generation are performed on the separated data. The encoding and vector generation scheme can be, for example, a hot vector, a digital representation, a binary representation, and a continuous representation. In addition, at operation 708, a combined representation of the encoded data is obtained.

[0102] FIG8 is a block diagram illustrating a classification fusion network (CaFuNet) of the emotion prediction manager 184 according to an embodiment described herein. Referring to FIG8, at operation 802, data is available, at operation 804, classified data is obtained, and at operation 806, classified features are obtained. In addition, at operation 808, the data is divided into images, text, audio, and video. The data is preprocessed and passed through a corresponding DNN model to obtain classified features (operation 810). At operation 812, the classified features obtained at both operations 806 and 810 are combined to obtain features. At operation 814, the combined data is passed through an autoencoder network (ie, a first network) 184a and a graph isomorphism network (ie, a second network) 184b. The outputs from the two networks are combined using an adder and passed through a fully connected layer (operation 816) to predict the user's emotions (operation 818).

[0103] FIG. 9 is a block diagram illustrating a CaFuNet having a user comprehension model of the emotion prediction manager 184 according to embodiments herein.

[0104] 9 , various user comprehension models having various layers of the autoencoder network 184 a are provided.

[0105] FIG. 10 is a flow chart illustrating the function of a DNN like an autoencoder network 184a to capture semantic information according to an embodiment described herein. Referring to FIG. 10 , at operation 1002 , normalized data is generated, and at operation 1004 , first data is received by the electronic device 100 . At operation 1006 , the first data is split according to the collection time window. At operation 1008 , the electronic device 100 determines whether the training phase is started, and if not, the normalized data is sent to the user understanding model 1012 and 1014 . If the training phase is started, at operation 1010 , the second data is received by the electronic device 100 . In addition, at operation 1016 , the adder receives data from operations 1006 , 1010 , and 1014 , and at operation 1018 , a combined representation is provided. The combined representation is then sent to the autoencoder network 184a (operation 1020 ). At operation 1022 , the model features representing the semantic learning features are projected in the dimensionality reduction space.

[0106] FIG. 11 is a block diagram illustrating an autoencoder network 184a according to an embodiment described herein. Model features representing semantic learning features are projected in a dimensionality reduction space. The autoencoder network 184a is capable of reducing the dimensionality of a large number of data features. As the number of features decreases, the autoencoder network 184a inherently reduces the training and inference time of subsequent models, and enables subsequent layers to converge faster. The autoencoder network 184a learns the semantic representation of the input data, thereby helping subsequent layers to better model the pattern.

[0107] The autoencoder network 184a embeds correlations between mutually related features and projects the input features in N dimensions such that similar data points are close and vice versa.

[0108] For the purpose of this disclosure, the term "input data" means first data: including non-private data of user activities on the user device (per collection time window); the term "second data" means data including demographic characteristics and lifestyle of the user from the user device. For the purpose of this disclosure, the term "training phase" refers to the collection of raw user data (age, gender, personality, etc.), and the term "inference phase" means the output of the missing second data and the user understanding model of the second data.

[0109] FIG. 12 is a block image showing that a DNN according to an embodiment described herein has the same function as a graph isomorphism network (GIN) 184b as a graph convolutional network (GCN) 184c for capturing structural and sequential information. Referring to FIG. 12 , a model feature representing structural and sequential information of data is provided. GIN 184b is used because data can be better structured and interpreted as a graph. There are inherent interrelationships in the first data (user activity and operation state data) obtained from within and across users, and therefore can be represented as nodes of a graph.

[0110] Sequential information (temporal information) and second data (representing demographic characteristics and lifestyles of users from user devices and / or devices connected to user devices) can be better represented structurally through the connection of graphs. GNN can process any type of graph. GIN 184b maximizes the representation of nodes (through better aggregation). Here, the nodes are the first data from the user (each collection time window), and the edges are of two types: intra-user and inter-user. "Intra-user" refers to the previous 'N' and next 'M' consecutive windows (a total of 'N' + 'M' intra-user edges), such as temporal features, timing information. "Inter-user" refers to selecting 'L' closest (measured by similarity), such as similarity of personality.

[0111] FIG. 13 is a flow chart illustrating a graph operation according to an embodiment of the present invention. Referring to FIG. 13 , at operation 1302, normalized data is generated. At operations 1304 and 1306, the electronic device 100 receives first data and second data, respectively. At operation 1308, the electronic device 100 determines split data according to a collection time window. At operation 1310, the split data and second data are provided to an adder; and at operation 1312, sequence information is obtained. At operation 1316, the user inner edge selector obtains selection criteria from operation 1314.

[0112] At operation 1318, the electronic device 100 determines whether the training phase is started. In response to determining that the training phase is not started, the second data is provided to the user understanding model (operation 1320). At operation 1322, a similarity measure is determined. At operation 1326, the user-to-user edge selector obtains the selection criteria from operation 1324. At operation 1328, a graph edge is defined, and at operation 1330, a graph representation is obtained. At operation 1332, a model feature representing the structure and order information of the data is obtained.

[0113] FIG. 14 illustrates an example of an insight level experience provided based on predicted user emotions according to embodiments herein.

[0114] 14 , at operation 1401 , the electronic device 100 provides a user with an emotional profile based on predicted emotions over a period of time, such as emotional statistics and happiness enhancement, emotional health, emotional habit formation over time (daily, weekly, monthly, etc.).

[0115] At operation 1402, the electronic device 100 provides insights to the user, such as the user's contacts, whether the user is happy based on a conversation or call, which application the user is very happy and / or sad about, etc. This can be used to decide the type of application that can be installed or uninstalled accordingly.

[0116] 15A-15D illustrate various examples of personalization of the electronic device 100 based on predicted user emotions according to embodiments described herein.

[0117] Referring to FIG. 15A , a prioritization of favorite contacts is shown. Referring to operation 1501a , consider an example where the predicted user emotion is “surprise.” And, when the user is surprised, the user tends to talk to close friends. Therefore, the contact list is modified accordingly. Referring to operation 1502a , consider an example where the predicted user emotion is “fear.” And, when the user is afraid, the user tends to talk to family members; therefore, the contact list is modified to display the contacts of family members at the top.

[0118] Referring to FIG. 15B , search result prioritization is shown. At operation 1501b, consider an example where the predicted user emotion is “happy”. And when the user is happy, the user tends to share social media status. Therefore, the social network site application is prioritized over other applications and presented to the user. At operation 1502b, consider an example where the predicted user emotion is “sad”. And when the user is sad, the user tends to discuss with friends, and therefore, the chat application is prioritized and presented to the user.

[0119] 15C , at operation 1501c , a conventional calendar application is provided. At operation 1502c , calendar sticker personalization is performed based on the predicted user emotion.

[0120] Referring to FIG. 15D , at operation 1501d , when the user attempts to reply from the notification panel, a conventional reply option is provided. At operation 1502d , an emotionally expressive intelligent reply suggestion based on emotion is provided in the proposed method. In addition, an adaptive notification UI (color and style) based on the emotion of the text can be adopted to prioritize between confusing notifications. Similarly, it can be used for adaptive color palettes and dialog tones, etc.

[0121] 16A-16D illustrate examples of using predicted emotions to create a digital twin in a metaverse according to embodiments of this document.

[0122] Referring to FIG. 16A , at operation 1601a, consider that the electronic device 100 predicts that the user's emotion is "happy". Then, at operation 1602a, the electronic device 100 replicates the predicted user emotion (i.e., "happy") in the virtual setting. At operation 1603a, the electronic device 100 defines a custom behavior of the user according to the predicted emotion in the virtual setting. Therefore, in the proposed method, the predicted user emotion is reflected in the virtual setting in both behavior and avatar.

[0123] Similarly, in Figure 16B, the digital twin (i.e., avatar) of the user in the virtual setting changes based on the predicted user emotion. Figure 16B provides various examples of avatars for replicating emotions predicted by the electronic device 100 (such as, for example, anxiety, sadness, happiness, disgust, anger, etc.).

[0124] Referring to FIG16C , the digital twin is used to provide posture personalization based on the predicted user emotions in the virtual setting. Similarly, the digital twin can be used to provide clothing preferences to the user based on the predicted emotions in the virtual setting. Here, clothing preferences or recommendations can be prioritized based on the predicted user emotions in the personal space. In addition, clothing recommendations in the store can be prioritized based on the user's personality.

[0125] Referring to FIG. 16D , digital twins can be used for emotion replication across virtual environments. Consider at operation 1601d that the predicted user emotion is “happy.” Then, at operation 1602d , the predicted user emotion is replicated across the time domain (i.e., based on the emotional behavior of a particular age range). At operation 1603d , the predicted user emotion is replicated across the character domain (i.e., based on the emotional behavior of a user-selected character (such as, for example, a dog, a human caricature, etc.)).

[0126] FIG17A shows an example environment modeling scenario according to an embodiment described herein. Referring to FIG17A , the example provides a scenario of environment modeling, where the environment changes with the mood of the user in the metaverse. Here, consider that the user suffers from depression when encountering rainy and cloudy weather. Therefore, the proposed method takes into account the user's environment and modifies the house features based on the user's mood (e.g., light, exterior, interior, etc.). In addition, the environment modification near the avatar area can also be provided based on the user's mood.

[0127] FIG. 17B illustrates another example environment modeling scenario according to embodiments herein.

[0128] Referring to FIG. 17B , this example provides a scenario for modeling a companion role for providing emotional support to a user. Here, a companion can be any object, such as, but not limited to, an animal, a person / friend, a bird, a robot, etc. Generally, animals are considered to provide happiness and companionship to humans. Animals can also provide emotional support to humans with mental health problems. At operation 1701, various emotions of the user predicted by the electronic device 100 are used to determine a virtual companion, such as a dog, that complements each emotion of the user (operation 1702). The virtual companion is used by the user to improve mental health and enhance positive emotions in the virtual environment (operation 1703).

[0129] Furthermore, the proposed method for predicting the user's emotions based on the state of the electronic device 100, the user's life pattern, and environmental factors can be used to provide emotional safety to the user in a virtual environment in a scenario like talking with emotions.

[0130] One such technique for providing emotional safety includes emotional masking. When a user is talking in a virtual environment, various user personality parameters, such as the user's pitch, tone, and language, can be modified and presented in the virtual environment according to the user's emotional state.

[0131] Another example of providing emotional safety to the user includes selectively showing or hiding the user's emotions based on the individuals with whom the user interacts in the virtual environment. Considering that the user is in a meeting, the user's emotions may be hidden from some attendees, or different emotions may be displayed to friends, coworkers, family, etc. For example, consider a situation where the user is attending an office meeting. In this case, the avatar may express emotions differently in formal and informal settings, such as, for example, tempered emotions in formal settings and raw emotions in informal settings.

[0132] Another scenario is to provide emotional safety in a virtual environment. Distance alone cannot solve this problem. There are other aspects such as detecting explicit content such as nudity and vulgar gestures and reviewing them to prevent user emotional stress. In addition, the proposed method can also be used to provide emotional privacy by revealing emotions based on environmental vulnerability, etc. This is particularly useful when children are involved, as children are sensitive to strong emotions (such as, for example, child-safe content).

[0133] The foregoing description of the specific embodiments will so fully reveal the general nature of the embodiments herein that others can easily modify and / or adapt these specific embodiments for various applications by applying current knowledge without departing from the general concepts, and therefore, these adaptations and modifications should and are intended to be understood to be within the meaning and scope of equivalents of the disclosed embodiments. It should be understood that the wording or terminology employed herein is for descriptive and not limiting purposes. Therefore, although the embodiments herein have been described according to preferred embodiments, those skilled in the art will recognize that the embodiments herein can be practiced by modification within the scope of the embodiments as described herein.

Claims

1. A method for predicting a user's emotions by an electronic device, the method include: receiving, by the electronic device, a user context, a device context, and an environmental context, wherein the user context, the device context, and the environmental context are collected by at least one of the electronic device and at least one of one or more other electronic devices connected to the electronic device; determining, by the electronic device, a combined representation of the user context, the device context, and the environmental context; determining, by the electronic device, a plurality of user characteristics based on the combined representation of the user context, the device context, and the environmental context; and The emotion of the user is predicted by the electronic device based on the plurality of user characteristics and the combined representation of the user context, the device context, and the environmental context.

2. The method according to claim 1, further comprising: include: The electronic device performs at least one of the following based on the predicted emotion of the user: modifying a user experience on the electronic device and on at least one other electronic device of the one or more other electronic devices, personalizing content on the electronic device and on at least one of the one or more other electronic devices, utilizing an emotion profile on the electronic device and on at least one of the one or more other electronic devices, generating at least one object for providing emotional support to the user, providing safety functions to the user in the virtual environment, and At least one user parameter in the virtual environment is modified.

3. The method according to claim 1, further comprising: include: determining, by the electronic device, at least one of: consumption of content by the user, unusual usage patterns on the electronic device or on at least one of the one or more other electronic devices, recurring activities performed by the user on the electronic device or on at least one of the one or more other electronic devices, and duration of time spent by the user on the electronic device or on at least one of the one or more other electronic devices; as well as The electronic device determines a quality of the predicted user emotion, wherein the quality of the predicted emotion is a positive emotion or a negative emotion.

4. The method according to claim 1, in, Determining, by the electronic device, the plurality of user characteristics based on a combined representation of the user context, the device context, and the environmental context comprises: providing, by the electronic device, the combined representation of the user context, the device context, and the environmental context to a first network and a plurality of intermediate models; and The plurality of user characteristics are determined by the electronic device.

5. The method according to claim 4, further comprising: include: A first intermediate set of emotions is predicted by the electronic device based on the plurality of user characteristics and the combined representation of the user context, the device context, and the environmental context.

6. The method according to claim 4, further comprising: include: providing, by the electronic device, the combined representation of the user context, the device context, and the environmental context to a second network and a third network; determining, by the electronic device, a local map sentiment prediction from the second network and a global node prediction from the third network; combining, by the electronic device, the local map emotion prediction and the global node prediction based on a specific weight; as well as A second intermediate emotion set is predicted by the electronic device.

7. The method according to claim 1, in, Determining, by the electronic device, a combined representation of the user context, the device context, and the environmental context includes: Determining, by the electronic device, a plurality of features associated with the user based on the user context, the device context, and the environmental context; classifying, by the electronic device, the plurality of features associated with the user into a plurality of categories corresponding to specific durations; generating, by the electronic device, at least one vector representation for each of the plurality of categories using the encoding; and The combined representation of the user context, the device context, and the environmental context is determined by the electronic device based on the at least one vector representation for each category of the plurality of categories.

8. The method according to claim 1, in, Predicting, by the electronic device, the user's emotion based on the user context, the device context, a combined representation of the environmental context, and the plurality of user characteristics comprises: receiving, by at least one second model of the electronic device, a first set of intermediate emotions and a second set of intermediate emotions; receiving, by the at least one second model of the electronic device, a classification clustering map; performing, by the at least one second model of the electronic device, an integration technique on the first intermediate emotion set and the second intermediate emotion set based on the classification cluster map; and The electronic device predicts the user's emotion.

9. The method according to claim 1, in, The plurality of user characteristics are determined using at least one first model, and wherein an emotion of the user is predicted using at least one second model.

10. A method for predicting a user's emotions by using an electronic device, the method include: receiving, by the electronic device, first data including user activity, an operating state of the electronic device, and an operating state of at least one other electronic device among one or more other electronic devices connected to the electronic device; receiving, by the electronic device, second data representing demographic characteristics and lifestyle of the user, wherein the second data is collected from at least one of the electronic device and at least one of the one or more other electronic devices connected to the electronic device; Normalizing the first data and the second data by the electronic device to input into a plurality of models; predicting, by the electronic device, a plurality of user characteristics from the model; as well as The emotion of the user is predicted by the electronic device based on the first data, the second data and the plurality of user characteristics.

11. An electronic device for predicting a user's emotions, include: at least one memory configured to store at least one instruction; at least one processor in communication with the at least one memory; as well as a communicator, the communicator communicating with the at least one memory and the at least one processor, Wherein, the at least one processor is configured to execute the at least one instruction to: receiving a user context, a device context, and an environmental context, wherein the user context, the device context, and the environmental context are collected by at least one of the electronic device and at least one of one or more other electronic devices connected to the electronic device; determining a combined representation of the user context, the device context, and the environmental context; determining a plurality of user characteristics based on the combined representation of the user context, the device context, and the environmental context; and The emotion of the user is predicted based on the plurality of user characteristics and the combined representation of the user context, the device context, and the environmental context.

12. An electronic device for predicting a user's emotions, include: at least one memory configured to store at least one instruction; at least one processor in communication with the at least one memory; as well as a communicator in communication with the at least one memory and the at least one processor, Wherein, the at least one processor is configured to execute the at least one instruction to: receiving first data, the first data comprising user activity, an operating state of the electronic device, and an operating state of at least one other electronic device among one or more other electronic devices connected to the electronic device; receiving second data representing demographic characteristics and lifestyle of the user, wherein the second data is collected from at least one of the electronic device and at least one of the one or more other electronic devices connected to the electronic device; normalizing the first data and the second data for input into a plurality of models; predicting a plurality of user characteristics from the model; and The emotion of the user is predicted based on the first data, the second data, and the plurality of user characteristics.