Method and system for providing interactive interface
Through deep learning algorithms and virtual assistant interfaces, user voice data is analyzed to identify emotional states, solving the problem of insufficient emotional recognition and interaction experience in the prior art, and realizing personalized user emotional interaction experience.
Patent Information
- Application Number
- CN202410973197.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2018-09-27
- Filing Date
- 2019-09-25
- Publication Date
- 2024-12-03
AI Technical Summary
The prior art is difficult to effectively imitate human brain functions, especially in terms of emotional recognition and interaction, and cannot provide a personalized user emotional interaction experience.
By using deep learning algorithms, combined with a virtual assistant interface, user voice data is analyzed to identify emotional states, and interaction types are determined based on emotional information, providing personalized notifications and responses.
Accurate recognition and personalized interaction of user emotions are achieved, user experience is improved, and appropriate responses can be provided according to the user's emotional state and event progress stage.
Smart Images

Figure CN119065497A_ABST
Abstract
Description
[0001] This application is a divisional application of the invention patent application with the application date of September 25, 2019, application number 201980004767.2, and invention name “Method and system for providing interactive interface”. Technical Field
[0002] The present disclosure relates to an artificial intelligence (AI) system and its application for mimicking the functions (e.g., recognition and determination) of the human brain by using a machine learning algorithm (such as deep learning). Specifically, the present disclosure relates to a method and apparatus for providing an interactive interface related to an event by using an AI system. Background Art
[0003] Artificial intelligence (AI) systems are systems that can achieve human-level intelligence. They are systems that can autonomously learn, determine, and become more intelligent, unlike existing rule-based intelligent systems. As AI systems iterate, the recognition rate increases and the user's preferences can be understood more accurately, so deep learning-based AI systems gradually replace existing rule-based intelligent systems.
[0004] AI technologies include machine learning (or deep learning) and element technologies that use machine learning.
[0005] Machine learning refers to algorithms used to classify / learn features of input data, and element technology refers to technology that mimics the functions of the human brain (e.g., recognition and determination) by using machine learning algorithms (such as deep learning), and includes technical fields such as language understanding, visual understanding, reasoning / prediction, knowledge representation, and operational control.
[0006] Examples of various fields to which AI technology is applicable are described below. Language understanding refers to technologies for recognizing and applying / processing human languages / characters, including natural language processing, machine translation, dialogue systems, query and response, speech recognition / synthesis, etc. Visual understanding refers to technologies for recognizing and processing objects like human vision, and includes object recognition, object tracking, image search, human recognition, scene understanding, spatial understanding, image enhancement, etc. Reasoning / prediction refers to technologies for determining information and logically performing reasoning and prediction, and includes knowledge / probability-based reasoning, optimized predictions, preference-based plans, recommendations, etc. Knowledge representation refers to a technology that automatically processes human experience information into knowledge data, and includes knowledge construction (data generation / classification), knowledge management (data utilization), etc. Operational control refers to technologies for controlling the autonomous driving of vehicles and the movement of robots, and includes motion control (e.g., navigation, collision avoidance or driving control), manipulation control (action control), etc. Summary of the invention
[0007] A method and system are provided for providing emotional interaction to a user based on the user's emotional information related to a user event.
[0008] Specifically, a method for providing a virtual assistant interface performed by an electronic device is provided, the method comprising: querying a user about a schedule by using a virtual assistant interface; receiving a response to the query from the user, the response comprising information about at least one event related to the schedule, wherein the receiving of the response to the query comprises receiving voice data of a user's speech as a response to the query; understanding the user's language by analyzing the response to the query to identify from the voice data at least one event related to the schedule and the user's emotional state to the at least one event related to the schedule; learning the user's emotional information to the at least one event related to the schedule based on the user's emotional state to the at least one event related to the schedule, wherein the emotional information comprises at least two or more complex emotions; determining an interaction type of the at least one event related to the schedule by obtaining an interaction type corresponding to the user's emotional information to the at least one event related to the schedule; and providing notification information to the at least one event related to the schedule through the virtual assistant interface based on the interaction type. BRIEF DESCRIPTION OF THE DRAWINGS
[0009] Figure 1 is a schematic diagram of a system for providing event-emotion based responses, according to an embodiment.
[0010] Figure 2 is a flowchart of a method for providing an interactive interface performed by an electronic device according to an embodiment.
[0011] Figure 3 is a schematic diagram for describing the operation of generating an event-emotion based response model (or artificial intelligence (AI) model) through deep learning according to an embodiment.
[0012] Figure 4 is a flowchart of a method of registering a new event based on voice data of a user's utterance according to an embodiment.
[0013] Figure 5 is a schematic diagram for describing operations performed by an electronic device in response to a user's utterance according to an embodiment.
[0014] Figure 6 is a schematic diagram for describing an operation of extracting a user event registered in an event database (DB) performed by an electronic device according to an embodiment.
[0015] Figure 7 is a flowchart of a method for acquiring user emotion information based on a progress stage of a user event, performed by an electronic device according to an embodiment.
[0016] Figure 8 is a schematic diagram for describing an operation performed by an electronic device to provide a notification or response based on a progress stage of a user event according to an embodiment.
[0017] Fig. 9 is a schematic diagram for describing an operation performed by an electronic device to obtain a user's emotion information based on a person, place, or time related to a user event according to an embodiment.
[0018] Fig.10 and Fig.11 is a schematic diagram for describing the operation of a personalized event-emotion DB according to an embodiment.
[0019] Fig.12 is a schematic diagram for describing an operation performed by an electronic device to modify default emotion information based on biometric information according to an embodiment.
[0020] Fig.13 is a flowchart of a method for determining a tone based on a user's emotion information according to an embodiment.
[0021] Fig.14 is a table for describing a tone determined based on emotion information of a user according to an embodiment.
[0022] Fig.15 is a schematic diagram for describing an operation performed by an electronic device to provide a notification by using a color corresponding to emotion information of a user according to an embodiment.
[0023] Fig.16 and Fig.17 is a schematic diagram for describing an operation of providing a diary interface performed by an electronic device according to an embodiment.
[0024] Fig.18 and Fig.19 is a block diagram of an electronic device according to an embodiment. DETAILED DESCRIPTION
[0025] Best Practices
[0026] According to an embodiment, a method for providing a virtual assistant interface performed by an electronic device includes: querying a user for schedule information by using the virtual assistant interface; receiving a response to the query from the user, the response including information about at least one event; identifying the user's emotional state for at least one event by analyzing the response to the query; based on the user's emotional state, learning the user's emotional information for at least one event; based on the user's emotional information, determining an interaction type of at least one event; and providing notification information for at least one event through the virtual assistant interface according to the interaction type.
[0027] The method for providing a virtual assistant interface performed by an electronic device may include: receiving voice data of a user's speech as a response to a query; extracting information related to at least one event from the voice data; and registering at least one event as a new event in the electronic device by using the extracted information.
[0028] The method for providing a virtual assistant interface performed by an electronic device may include: obtaining default emotion information related to at least one event from a table in which events are mapped to emotion information; identifying a user's emotional state with respect to at least one event based on information obtained by analyzing voice data; and learning the user's emotional information with respect to at least one event by modifying the default emotion information based on the user's emotional state.
[0029] The method for providing a virtual assistant interface performed by an electronic device may include: determining the progress stage of at least one event by comparing the current time with the schedule time of at least one event; and optimizing the user's emotional information about the at least one event based on the progress stage of the at least one event.
[0030] The method for providing a virtual assistant interface performed by an electronic device may include: obtaining feedback information about at least one event from a user after at least one event is completed; and refining the user's emotional information about at least one event based on the obtained feedback information.
[0031] The method for providing a virtual assistant interface performed by an electronic device may include learning a user's emotion information by using information related to at least one of a person related to at least one event, a schedule time of at least one event, and a place related to at least one event.
[0032] The method for providing a virtual assistant interface performed by an electronic device may include determining a tone for providing notification information of at least one event based on emotion information of a user.
[0033] The method for providing a virtual assistant interface, performed by an electronic device, may include: selecting a color corresponding to emotion information of a user; and displaying notification information of at least one event by using the selected color.
[0034] The method for providing a virtual assistant interface performed by an electronic device may include providing a response to a user utterance related to at least one event based on the determined interaction type.
[0035] The method for providing a virtual assistant interface performed by an electronic device may include: considering a user's condition, determining a notification providing method related to at least one event; and providing notification information for at least one event based on the determined notification providing method.
[0036] The method for providing a virtual assistant interface performed by an electronic device may include providing a diary interface including emotion information of a user related to at least one event.
[0037] According to another embodiment, an electronic device includes: an outputter configured to provide a virtual assistant interface; and at least one processor configured to: query a user for schedule information through the virtual assistant interface; receive a response to the query from the user, the response including information about at least one event; identify the user's emotional state towards at least one event by analyzing the response to the query; based on the user's emotional state, learn the user's emotional information towards at least one event; based on the user's emotional information, determine an interaction type of at least one event; and provide notification information for at least one event through the virtual assistant interface according to the interaction type.
[0038] According to another embodiment, a computer program product includes a recording medium having recorded thereon a program for executing a method, the method comprising: querying a user for schedule information through a virtual assistant interface; receiving a response to the query from the user, the response comprising information about at least one event; identifying the user's emotional state towards at least one event by analyzing the response to the query; based on the user's emotional state, learning the user's emotional information towards at least one event; based on the user's emotional information, determining an interaction type of at least one event; and providing notification information for at least one event through the virtual assistant interface according to the interaction type.
[0039] According to another embodiment, a method for providing a virtual assistant interface performed by an electronic device includes: providing notification information of at least one event to a user through the virtual assistant interface, obtaining feedback information of the user on at least one event included in the notification information, identifying the user's emotional state for at least one event by analyzing the user's feedback information, and learning the user's emotional information for at least one event based on the user's emotional state.
[0040] According to another embodiment, an electronic device includes: an outputter configured to provide a virtual assistant interface; and at least one processor configured to provide notification information of at least one event to a user through the virtual assistant interface, obtain feedback information of the user on at least one event included in the notification information, identify the user's emotional state for at least one event by analyzing the user's feedback information, and learn the user's emotional information for at least one event based on the user's emotional state.
[0041] According to another embodiment, a computer program product includes a recording medium having recorded thereon a program for executing a method, the method comprising: providing notification information of at least one event to a user through a virtual assistant interface, obtaining feedback information of the user on at least one event included in the notification information, identifying the user's emotional state for at least one event by analyzing the user's feedback information, and learning the user's emotional information for at least one event based on the user's emotional state.
[0042] According to another embodiment, a method for providing an interactive interface performed by an electronic device includes: obtaining emotion information of a user related to a user event registered in the electronic device, determining an interaction type corresponding to the user event based on the emotion information of the user, and providing a response to a user utterance related to the user event based on the determined interaction type.
[0043] According to another embodiment, an electronic device includes: an outputter configured to provide an interactive interface; at least one processor; and a memory storing at least one instruction executed by the at least one processor to instruct the electronic device to obtain emotion information of a user related to a user event registered in the electronic device, determine an interaction type corresponding to the user event based on the emotion information of the user, and provide a response to a user utterance related to the user event through the interactive interface based on the determined interaction type.
[0044] According to another embodiment, a computer program product includes a recording medium having recorded thereon a program for executing a method, the method comprising: obtaining emotion information of a user related to a user event, determining an interaction type corresponding to the user event based on the emotion information of the user, and providing a response to a user utterance related to the user event based on the determined interaction type.
[0045] Before describing the present disclosure in detail, the terms used in the specification will now be briefly described.
[0046] Although the terms used in this document are selected from the currently widely used general terms as much as possible in consideration of the functions obtained according to the present disclosure, these terms may be replaced by other terms based on the intentions of ordinary technicians in the field, habits, the emergence of new technologies, etc. In certain cases, terms arbitrarily selected by the applicant may be used, and in this case, the meanings of these terms may be described in the corresponding parts of the detailed description. Therefore, it should be noted that the terms used in this document are interpreted based on their actual meanings and the entire contents of this specification, rather than being interpreted based only on the names of the terms.
[0047] It will be understood that when used in this document, the terms "comprises" and / or "includes" specify the presence of the elements described, but do not exclude the presence or addition of one or more other elements. The terms "... unit" or "... module" are used to represent an entity for performing at least one function or operation, and can be implemented using hardware, software, or a combination of hardware and software.
[0048] Hereinafter, the present disclosure will be described in detail by explaining the embodiments of the present disclosure with reference to the accompanying drawings. However, the present disclosure can be implemented in many different forms and is not interpreted as being limited to the embodiments of the present disclosure set forth herein. In the accompanying drawings, for the sake of clarity, parts not related to the embodiments of the present disclosure are not shown, and similar reference numerals always represent similar elements.
[0049] Figure 1 is a schematic diagram of a system for providing event-emotion based responses, according to an embodiment.
[0050] Reference Figure 1 According to an embodiment, the system for providing an event-emotion based response may include an electronic device 1000. According to an embodiment, in addition to the electronic device 1000, the system for providing an event-emotion based response may further include a server (not shown) and a wearable device for measuring a user's biometric signal. Fig.12 DETAILED DESCRIPTION
[0043] A system for providing event-emotion based responses includes an embodiment of an electronic device 1000 and an external wearable device.
[0051] The electronic device 1000 according to the embodiment can be implemented in various forms. For example, as used herein, the electronic device 1000 may include a digital camera, a smart phone, a laptop computer, a tablet PC, an e-book reader, a digital broadcast receiver, a personal digital assistant (PDA), a portable multimedia player (PMP), a navigation system or an MP3 player, but is not limited thereto. As used herein, the electronic device 1000 may be a wearable device that can be worn by a user. The wearable device may include an accessory device (e.g., a watch, a ring, a wristband, an ankle band, a necklace, glasses or contact lenses), a head-mounted device (HMD), a fabric or clothing integrated device (e.g., an electronic garment), a body-attachable device (e.g., a skin pad) and an implantable body device (e.g., an implantable circuit), but is not limited thereto. In the following description, for ease of explanation, the case where the electronic device 1000 is a smart phone will be described as an example.
[0052] According to an embodiment, the electronic device 1000 may be a device for providing an interactive interface. The interactive interface may be a user interface for receiving input (e.g., voice input or text input) from a user and providing a response to the input from the user. According to an embodiment, the interactive interface may include, for example, a virtual assistant or an artificial intelligence (AI) assistant, but is not limited thereto. A virtual assistant or an AI assistant may be a software agent for processing tasks required by a user and providing specialized services to the user.
[0053] According to an embodiment, the electronic device 1000 can obtain (or analyze or infer) the emotional state of the user related to the user event 100 by using an AI model, and adaptively provide a response through an interactive interface based on the emotional state of the user. For example, when the emotional state of the first user related to the user event 100 is a fear state 10, the electronic device 1000 can determine a first interaction type and respond to the first user's speech based on the first interaction type. When the emotional state of the first user related to the user event 100 is an angry state 20, the electronic device 1000 can determine a second interaction type and respond to the first user's speech based on the second interaction type. When the emotional state of the first user related to the user event 100 is a happy state 30, the electronic device 1000 can determine a third interaction type and respond to the first user's speech based on the third interaction type. Here, the first to third interaction types can be different types, so the electronic device 1000 can provide event-based emotional interaction to the user.
[0054] As used herein, user events 100 may be events related to a user, for example, events detected by the electronic device 1000 during a conversation with the user, or events pre-registered by the user in, for example, a schedule application, a calendar application, a contact list application, or a social networking service (SNS) application. According to an embodiment, user events 100 may include, for example, work-related events (e.g., meetings, presentations, or business trips), events related to acquaintances (e.g., a friend's wedding, a family member's birthday party, or a wedding anniversary), events related to hobbies (e.g., watching sports games, watching movies, booking tickets for music performances, photography club activities, participating in tennis matches, or participating in marathons), events related to celebrities (e.g., concerts or fan meetings), or official events (e.g., presidential elections, the World Cup, or Christmas), but are not limited thereto.
[0055] A processor and a memory are used to perform AI-related functions according to the present disclosure. The processor may include one or more processors. In this case, the one or more processors may be a general-purpose processor (such as a central processing unit (CPU), an application processor (AP), and a digital signal processor (DSP)), a dedicated graphics processor (such as a graphics processing unit (GPU) and a visual processing unit (VPU)) or a dedicated AI processor (such as a digital processing unit (NPU)). One or more processors control the processing of input data based on predefined operating rules or AI models stored in the memory. Optionally, when one or more processors are dedicated AI processors, the dedicated AI processor can be designed with a hardware structure dedicated to processing a specific AI model.
[0056] The predefined operating rules or AI models are formulated by training. In this article, formulating by training means training the basic AI model based on multiple training data using a learning algorithm, thereby formulating predefined operating rules or AI models configured to achieve the desired characteristics (or purposes). The training can be performed by a device with AI functions according to the present disclosure, or by a separate server and / or system. The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited thereto.
[0057] The AI model may include multiple neural network layers. Each of the multiple neural network layers has multiple weight values, and the neural network calculation is performed by calculating between the calculation results of the previous layer and the multiple weight values. The multiple weight values of the multiple neural network layers can be optimized by training the results of the AI model. For example, multiple weight values can be modified to reduce or minimize the loss value or cost value obtained by the AI model during training. The artificial neural network may include, for example, a convolutional neural network (CNN), a deep neural network (DNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN) or a deep Q network, but is not limited thereto.
[0058] Therefore, according to the embodiment, events meaningful to the user can be detected through the dialogue, and active interaction or necessary information can be provided to the user based on the detected events, thereby providing real emotional interaction. Figure 2 The operations performed by the electronic device 1000 to provide emotional interaction based on the user's emotional state with respect to the user event 100 are described in detail.
[0059] Figure 2is a flowchart of a method for providing an interactive interface performed by the electronic device 1000 according to an embodiment.
[0060] In operation S210 , the electronic device 1000 may obtain emotion information of a user related to a user event registered in the electronic device 1000 .
[0061] According to an embodiment, the electronic device 1000 may check user events registered in an event database (DB). The event DB may include or be connected to, for example, a schedule application (e.g., a to-do list application), a calendar application, an SNS application, or a contact list application. The user event may be an event manually input by the user, or an event automatically input by the electronic device 1000 or a server.
[0062] According to an embodiment, the electronic device 1000 may obtain the user's emotional information related to the user event by analyzing the response from the user and learning the user's emotional information related to the user event. For example, the electronic device 1000 may query the user for schedule information by using a virtual assistant interface. In this case, the electronic device 1000 may receive a response including information about at least one event from the user. The electronic device 1000 may identify the user's emotional state to at least one event by analyzing the response received from the user. The electronic device 1000 may learn the user's emotional information to at least one event. In this article, the learning of the user's emotional information to at least one event may mean determining the weight of at least one emotional element associated with at least one event.
[0063] According to an embodiment, the electronic device 1000 can extract a user event from the voice data of the user's utterance and register the extracted user event as a new event in the event DB. Figure 4 An operation of extracting a user event from voice data of a user utterance, performed by the electronic device 1000, is described in detail.
[0064] According to an embodiment, the user's emotion information related to the user event may be information about the user's emotional state to the user event. Here, the user's emotion information may be information indicating a single emotional state, or information indicating a complex emotional state including two or more emotions. For example, the emotion information may include at least one emotion of joy, tension, fear, happiness, fear, unhappiness, annoyance, embarrassment, sadness, frustration, regret, jealousy, disappointment, confidence, love, hatred, love and hate, fun, excitement, stability, relaxation, and thriller, but is not limited thereto.
[0065] According to an embodiment, the electronic device 1000 may obtain the user's emotion information related to the user event when the user speech related to the user event is detected. According to another embodiment, the electronic device 1000 may obtain the user's emotion information on the user event to actively provide interaction before the user speech related to the user event. For example, the electronic device 1000 may check the schedule time of the user event and obtain the user's current emotion information on the user event to provide a notification related to the user event 10 minutes before the user event.
[0066] According to an embodiment, the electronic device 1000 may obtain the user's emotion information related to the user event by using an AI model. For example, the electronic device 1000 may obtain the user's emotion information corresponding to a specific user event from an event-emotion DB generated (or refined) by an AI model. The event-emotion DB may store the average emotion information of ordinary people for a specific event, or the emotion information personalized for each user for a specific event, but is not limited thereto.
[0067] According to an embodiment, the AI model may be trained based on at least one of the emotion information analyzed by the expert and the previous emotion information of the user. The AI model may be stored in the electronic device 1000 or in a server (e.g., a cloud server) connected to the electronic device 1000. Figure 3 Describe the AI model in detail.
[0068] According to an embodiment, the electronic device 1000 can determine the progress stage of the user event by comparing the current time with the scheduled time of the user event. Here, the current time may include the time when the speech is detected from the user, but is not limited to this. For example, the electronic device 1000 can determine whether the current time is the time before the user event occurs or the time after the user event is completed. Taking into account the progress stage of the user event, the electronic device 1000 can obtain the user's emotional information related to the user event. For example, when the user event is to participate in a marathon, before the user event occurs, the electronic device 1000 can determine the user's emotional information as "tension weight: 0.5, joy weight: 0.5". On the other hand, after the user event (e.g., participating in a marathon) ends, the electronic device 1000 can determine the user's emotional information as "happy weight: 0.3, joy weight: 0.7".
[0069] According to an embodiment, the electronic device 1000 may obtain the user's emotion information by considering the additional information related to the user event. For example, the electronic device 1000 may subdivide the user's emotion information by considering at least one of the person related to the user event, the schedule time of the user event, and the place related to the user event. In this case, the electronic device 1000 may obtain the additional information related to the user event by directly asking the user for the additional information or by parsing emails, messages, etc. Fig. 9 An operation performed by the electronic device 1000 to segment the emotion information of the user in consideration of the additional information related to the user event is described in detail.
[0070] According to an embodiment, the electronic device 1000 may obtain default emotion information related to the user event from a table in which events are mapped to emotion information. The default emotion information may be the user's previous emotion information related to the user event, or the average emotion information of ordinary people on the user event. The table in which events are mapped to emotion information may be stored in an event-emotion DB associated with the AI model, but is not limited thereto.
[0071] According to an embodiment, the electronic device 1000 may obtain the actual emotion information of the user related to the user event by obtaining the biometric information of the user (e.g., heart rate information, pulse information, breathing information, eyelid twitching information, facial expression change information, sweating information, body temperature change information, or voice input) and modifying the default emotion information related to the user event based on the biometric information of the user. For example, the electronic device 1000 may obtain the default emotion information related to at least one event from a table in which the event is mapped to the emotion information, identify the user's emotional state for at least one event based on the information obtained by analyzing the voice data of the user's speech, and learn the user's emotional information for at least one event by modifying the default emotion information based on the user's emotional state. The modification of the default emotion information may mean adjusting the weight corresponding to each emotion element included in the default emotion information. For example, when the default emotion information is "tension weight: 0.3, joy weight: 0.7", the electronic device 1000 may increase the tension weight from "0.3" to "0.6" based on the biometric information of the user, and reduce the joy weight from "0.7" to "0.4".
[0072] According to an embodiment, the electronic device 1000 may obtain the biometric information of the user by using a sensor included therein, or may obtain the biometric information of the user from an external device (e.g., an external wearable device). For example, the electronic device 1000 may obtain a facial image of the user by using at least one camera (e.g., an ordinary image sensor, an infrared sensor, or a depth sensor), and detect changes in the user's facial expression or eye twitching by analyzing the facial image of the user. In addition, the electronic device 1000 may detect changes in the user's body temperature by using an infrared sensor. In this case, the electronic device 1000 may predict the user's current emotional state based on the user's facial expression changes or eye twitching. In addition, the electronic device 1000 may receive voice data from the user through a microphone, and may predict the user's actual emotional state by analyzing the voice data. The following will refer to Fig.12 An operation performed by the electronic device 1000 to receive biometric information of a user from an external device is described in detail.
[0073] In operation S220, the electronic device 1000 may determine an interaction type corresponding to the user event based on the emotion information of the user.
[0074] Here, the interaction type refers to an interaction style used to provide a response to the user, and the range of additional information to be provided in addition to the information related to the user event, the tone used to provide the interaction, etc. may differ according to the interaction type. For example, the interaction type may include an encouragement type (e.g., change subject weight: 0.3, cheer up weight: 0.7), an empathy type (e.g., empathy weight: 0.9, cheer up weight: 0.1), a console type (e.g., console weight: 0.8, empathy weight: 0.2), or a cheer up type (e.g., cheer up weight: 0.9, provide additional information weight: 0.1), but is not limited thereto.
[0075] According to an embodiment, when the user's emotional information for the first event is "tension weight: 0.9" and "joy weight: 0.1", the electronic device 1000 may determine the interaction type corresponding to the first event as "encouragement type". When the user's emotional information for the second event is "joy weight: 0.9" and "tension weight: 0.1", the electronic device 1000 may determine the interaction type corresponding to the second event as "empathy type".
[0076] According to an embodiment, the electronic device 1000 may determine the interaction type corresponding to the user event by using an AI model. For example, the electronic device 1000 may determine the interaction type mapped to the emotion information based on the interaction method DB generated (or optimized) by the AI model. The interaction method DB may store a table in which the emotion information is mapped to the interaction type, but is not limited thereto.
[0077] According to an embodiment, the electronic device 1000 may determine the interaction type by reflecting the user's preference. For example, by reflecting the user's preference, the electronic device 1000 may determine information indicating whether to use a simple image (e.g., emoticon) to provide interaction, an interaction method (e.g., an interaction method using text, an interaction method using voice, or an interaction method using a notification window), or a voice color.
[0078] According to an embodiment, the electronic device 1000 may determine a tone for providing interaction with respect to the user's speech in consideration of the user's emotional information. In addition, the electronic device 1000 may determine a color for providing interaction with respect to the user's speech in consideration of the user's emotional information. Fig.14 and Fig.15 An operation of determining a tone or a color performed by the electronic device 1000 is described in detail.
[0079] In operation S230, electronic device 1000 can provide a response to a user's speech related to a user event based on the determined interaction type. Herein, providing a response to the user's speech can include an active interaction for causing the user's speech (e.g., providing a notification related to a user event).
[0080] According to an embodiment, the electronic device 1000 can provide a response to a user's speech related to a user event by using an AI model. For example, the electronic device 1000 can extract a response corresponding to the user's speech from a response DB generated (or optimized) by the AI model. The response DB can store a table in which interaction types are mapped to phrases that the user can input and response phrases, but is not limited thereto.
[0081] The case where the user inputs the first phrase will now be described as an example. The electronic device 1000 may select the first response phrase when the determined interaction type is the encouragement type, or select the second response phrase when the determined interaction type is the empathy type.
[0082] According to an embodiment, for different interaction types, different additional information may be used to select the same response phrase. Now, the case where the user inputs the first phrase is described as an example. When the determined interaction type is the encouragement type, the electronic device 1000 may select the first response phrase and select the first text as the additional information. Otherwise, when the determined interaction type is the empathy type, the electronic device 1000 may select the first response phrase and select the second text as the additional information.
[0083] According to an embodiment, the electronic device 1000 may provide a response in a text form or in a voice form on a chat window. The electronic device 1000 may provide a response in a text form on a notification window (eg, a pop-up window), but is not limited thereto.
[0084] According to an embodiment, the electronic device 1000 may select a color corresponding to the user's emotional information and display a response to the user's speech related to the user event by using the selected color. For example, when the first color is selected based on the user's first emotional information for the first event, the electronic device 1000 may provide a response on the notification window displayed in the first color. Otherwise, when the second color is selected based on the user's second emotional information for the second event, the electronic device 1000 may provide a response on the notification window displayed in the second color.
[0085] According to an embodiment, the electronic device 1000 can provide a response by applying the expression style preferred by the user. For example, when "You can do it" and "Go for it" can be provided as responses to the first phrase input by the user, the electronic device 1000 can select "You can do it" corresponding to the expression style preferred by the user.
[0086] The electronic device 1000 may select a user-preferred tone, a user-preferred voice, a user-preferred notification method, or a user-preferred color, but is not limited thereto.
[0087] According to an embodiment, the electronic device 1000 may consider the user's situation to perform active interaction. For example, the electronic device 1000 may determine a notification providing method related to a user event in consideration of the user's situation. The electronic device 1000 may provide notifications related to user events based on the determined notification providing method. For example, when a user is giving a lecture in the afternoon, the electronic device 1000 may display a notification (or reminder) related to an evening event on the screen without sound, and then provide the notification (or reminder) again in the form of vibration or sound when the current time is close to the scheduled time of the evening event.
[0088] According to an embodiment, the electronic device 1000 may provide a diary interface including the user's emotion information related to the user event. For example, the electronic device 1000 may provide a simple image or graphic indicating the user's emotion information at each progress stage of the user event. Fig.16 and Fig.17 An operation performed by the electronic device 1000 to provide a diary interface is described in detail.
[0089] Although the above combination Figure 2 The example of describes the case where the electronic device 1000 provides a response to the user's voice input, but the user input is not limited thereto. The electronic device 1000 may also provide a response to the user's text input based on the user's emotion information. Figure 3 Describe the AI model in detail.
[0090] Figure 3 is a schematic diagram for describing the operation of generating an event-emotion based response model (or AI model) through deep learning according to an embodiment.
[0091] Reference Figure 3 According to an embodiment, an AI processor included in the electronic device 1000 or the server may generate an event-emotion based response model 300 for determining an interaction method by training an artificial neural network. In the following description, for ease of explanation, the event-emotion based response model 300 may also be referred to as an AI model or a response model. "Training" of an artificial neural network may mean constructing a mathematical model by appropriately changing weights based on data, in which the connection of neurons in the artificial neural network can make the best decision.
[0092] According to an embodiment, the AI processor may obtain data created by an expert, published data, etc. as training data 301, and generate an event-emotion-based response model 300 by using the training data 301. The data created by the expert may include emotional information analyzed or extracted by the expert from a phrase predicted to be spoken by the user. The published data may include emotional information analyzed or extracted from a text collected from a network or SNS server. For example, the published data may include happy emotional information analyzed / extracted from the phrase "Skipped school today skipping school tomorrow haha". The data created by the expert or the published data may be converted into a preset format and thus used as training data 301 for generating an event-emotion-based response model 300.
[0093] According to an embodiment, the AI processor may obtain emotion information (e.g., feedback information) input from a user, emotion information obtained from the user's biometric information, etc. as personalized training data 302, and generate an event-emotion-based response model 300 by using the personalized training data 302. The emotion information (e.g., feedback information) input from a user or the emotion information obtained from the user's biometric information may be converted into a preset format and thus used as personalized training data 302 for generating an event-emotion-based response model 300.
[0094] According to an embodiment, the event-emotion based response model 300 may be trained using the training data 301 to generate the event-emotion DB 310, the interaction method DB 320, and the response DB 330. According to an embodiment, the event-emotion based response model 300 may optimize the event-emotion DB 310, the interaction method DB 320, and the response DB 330 by adding or refining the training data 301.
[0095] According to an embodiment, the event-emotion DB 310 is a DB that defines the correlation between events and emotions, and may include a table in which events are mapped to emotion information. For example, the event-emotion DB 310 may include information such as "first event: nervous (0.6), joy (0.4)" and "second event: fear (0.7), nervous (0.3)".
[0096] According to an embodiment, the event-emotion DB 310 may store a table that subdivides the emotion information based on the time, person, and place related to the event. For example, the event-emotion based response model 300 may subdivide the emotion information of the user in consideration of the time, person, and place related to the event. When detailed information about the time, person, and place related to the event is not stored in the event DB, the event-emotion based response model 300 may obtain the information by querying the user about the time, person, and place related to the event. Fig. 9 A table that describes the segmented sentiment information in detail.
[0097] According to an embodiment, the interaction method DB 320 is a DB that defines the correlation between emotions and interaction methods, and can store a table in which emotion information is mapped to interaction types. For example, the interaction method DB 320 may include information such as "first emotion information (nervous (0.6), joy (0.4))-first interaction type (change topic (0.3), encouragement (0.7))" and "second emotion information (relaxed)-second interaction type (empathy (0.9))".
[0098] According to an embodiment, the response DB 330 is a DB that defines the correlation between the interaction method (or interaction type) and the response phrase, and can store a table in which the interaction type is mapped to the response phrase. For example, the response DB 330 may include information such as "first interaction type (e.g., encouragement type: change topic (0.3), encouragement (0.7))-response phrase (Go for it (0.6) / You can do it (0.4))".
[0099] According to an embodiment, the event-emotion based response model 300 may obtain personalized training data 302 , and personalize the event-emotion DB 310 , the interaction method DB 320 , and the response DB 330 by using the personalized training data 302 .
[0100] According to an embodiment, after the user event is completed, the electronic device 1000 may obtain feedback about the user event from the user. The electronic device 1000 may input the feedback about the user event into the event-emotion based response model 300 as personalized training data 302. In this case, the event-emotion based response model 300 may optimize the table in which events are mapped to emotion information included in the event-emotion DB 310 based on the feedback of the user event. For example, when the electronic device 1000 has determined the user's emotion information for the first event as "nervous (0.5), joy (0.5)", but the feedback input from the user indicates that the user's emotion information is "nervous (0.8), joy (0.2)", the electronic device 1000 may modify the user's emotion information for the first event from "nervous (0.5, joy (0.5)" to "nervous (0.8, joy (0.2)" in the event-emotion DB 310.
[0101] According to an embodiment, the electronic device 1000 can obtain feedback on the user's emotional information before the user event occurs. In this case, the event-emotion-based response model 300 of the electronic device 1000 can optimize the emotional information of each progress stage of the user event, which is included in the event-emotion DB 310.
[0102] According to an embodiment, the electronic device 1000 may obtain biometric information of a user and personalize the event-emotion DB 310 by using actual emotion information of the user predicted based on the biometric information.
[0103] The electronic device 1000 may personalize the interaction method DB 320 and the response DB 330 by obtaining information on a user-preferred interaction type or a user-preferred expression style.
[0104] Now refer to Figure 4 and Figure 5A method performed by the electronic device 1000 to detect a new event from voice data of a user utterance and to provide a response to the user utterance related to the new event by using the event-emotion based response model 300 is described in detail.
[0105] Figure 4 is a flowchart of a method for registering a new event based on voice data of a user utterance according to an embodiment.
[0106] In operation S410, the electronic device 1000 may extract information related to a user event from voice data of a user utterance.
[0107] According to an embodiment, the electronic device 1000 may generate data based on natural language processing as a result of analyzing voice data. For example, the electronic device 1000 may convert voice data into text data based on natural language processing, and detect information related to user events from the text data. For example, the electronic device 1000 may detect "wedding" as a new user event by analyzing voice data stating "my friend Jenna is having a wedding this Saturday", and obtain "person getting married: Jenna, relationship: friend, wedding date: this Saturday (6 / 30)" as information related to a new user event (e.g., wedding).
[0108] In operation S420, the electronic device 1000 may register the user event as a new event by using the extracted information. For example, the electronic device 1000 may register the wedding as a new event by storing information related to the user event (e.g., wedding) detected from the voice data in the event DB connected to the AI model (e.g., "Person getting married: Jenna, Relationship: Friend, Wedding date: This Saturday (6 / 30)").
[0109] According to an embodiment, the event DB may be connected to, for example, a schedule application, a calendar application, a contact list application, or an SNS application, but is not limited thereto. The event DB is a DB that defines user events meaningful to the user, and may include information about events manually input by the user and information about events automatically registered by the system.
[0110] In operation S430, the electronic device 1000 may extract the user's emotion information from the voice data of the user's speech. For example, the electronic device 1000 may convert the voice data into text data based on natural language processing, and detect information related to the emotion state from the text data. For example, the electronic device 1000 may detect "nervous" as the user's emotion information by analyzing the voice data stating "I am nervous to sing at my friend Jenna's wedding this Saturday."
[0111] In operation S440, when the user's emotion information is not extracted from the voice data, the electronic device 1000 may obtain the user's emotion information related to the user event from the event-emotion DB. Operation S440 may correspond to Figure 2 Operation S220 is described below, and thus a detailed description thereof is not provided here.
[0112] In operation S450, the electronic device 1000 may determine an interaction type corresponding to the user event based on the emotion information of the user.
[0113] When the user's emotion information is extracted from the voice data of the user's utterance, the electronic device 1000 can determine the interaction type corresponding to the user event based on the extracted emotion information. When the user's emotion information is not extracted from the voice data of the user's utterance, the electronic device 1000 can determine the interaction type corresponding to the user event based on the user's emotion information obtained from the event-emotion DB.
[0114] According to an embodiment, the electronic device 1000 may determine the interaction type mapped to the emotion information based on the interaction method DB generated (or optimized) by the AI model. Operation S450 corresponds to Figure 2 Operation S220 is described below, and thus a detailed description thereof is not provided here.
[0115] In operation S460, the electronic device 1000 may provide a response to the user utterance related to the user event based on the determined interaction type.
[0116] According to an embodiment, the electronic device 1000 may extract a response mapped to an interaction type from a response DB generated (or optimized) by an AI model, and provide the extracted response to a user through an interaction interface. Operation S460 corresponds to Figure 2 Operation S230 is described below, and thus a detailed description thereof is not provided here.
[0117] Figure 5 is a schematic diagram for describing an operation performed by the electronic device 1000 in response to a user utterance according to an embodiment.
[0118] In operation S510, when the user speaks, the electronic device 1000 can obtain the user's voice data through the microphone and understand the user's language by analyzing the voice data. That is, the electronic device 1000 can extract the user's intention from the voice data, extract information related to the user's event, and extract emotional information about the user's event (S510). For example, when the user says "I have to make an important presentation in the afternoon", the electronic device 1000 can extract "register for a new event" as the user's intention and extract "presentation, afternoon" as information related to the new event. In this case, no emotional information may be extracted from the user's speech.
[0119] In operation S520, when the user's language is understood, the electronic device 1000 can determine the interaction. For example, the electronic device 1000 can register "Preaching, afternoon" as a new event in the event DB 340 according to the user's intention. In operation S521, the electronic device 1000 can extract the first emotion information corresponding to the preaching from the event-emotion DB 310 (for example, "tension weight: 0.6, joy weight: 0.4"). In operation S522, the electronic device 1000 can extract the first interaction method (for example, first interaction type: other information (0.3), cheer up (0.7)) corresponding to the first emotion information (for example, "tension weight: 0.6, joy weight: 0.4") from the interaction method DB 320.
[0120] In operation S530, the electronic device 1000 may set a response. For example, the electronic device 1000 may modify the response based on the user event and the interaction method. In operation S531, the electronic device 1000 may extract "You are preaching" as a default response to the user's speech from the response DB 330, and extract "Good luck" as an additional response. The electronic device 1000 may output "You are preaching. Good luck!" through the interaction interface.
[0121] Because "lecture, afternoon" is registered as a new event, the electronic device 1000 can monitor the progress of the "lecture". For example, the electronic device 1000 can obtain the user's emotion information before and after the lecture. The electronic device 1000 can optimize the personal emotion DB based on the obtained user's emotion information. For example, the electronic device 1000 can personalize the event-emotion DB 310 by modifying the default emotion information included in the event-emotion DB 310 (S540).
[0122] An operation performed by the electronic device 1000 to respond to a user based on the user's emotion information to a user event pre-registered in the event DB 340 will now be described.
[0123] Figure 6 is a schematic diagram for describing an operation of extracting a user event registered in the event DB 340, performed by the electronic device 1000, according to an embodiment.
[0124] The electronic device 1000 may receive the first voice data 601 based on the first utterance of the user 600. For example, the electronic device 1000 may receive a query input asking “What is my schedule today?” from the user 600. In this case, the electronic device 1000 may extract the intention of the user 600 (e.g., checking the schedule of the user 600 today) by analyzing the first voice data 601 received from the user 600.
[0125] The electronic device 1000 may access the event DB 340 connected to the event-emotion based response model 300 to check the schedule of the user 600 today according to the intention of the user 600. The electronic device 1000 may check the event information related to the user 600 (e.g., presentation to the vice president) from the event DB 340 and provide a first response 602. For example, the electronic device 1000 may provide a first response 602 stating that “the schedule for this afternoon is a presentation to the vice president”.
[0126] In this case, the electronic device 1000 may receive the second voice data 603 from the user 600 who has heard the first response 602. For example, the electronic device 1000 may receive the second voice data 603 stating “Oh, I see! I must do well…”. In order to provide a response to the second voice data 603, the electronic device 1000 may obtain the default emotion information (e.g., “tension weight: 0.6, joy weight: 0.4”) corresponding to the user event “Preaching to the Vice President” from the event-emotion DB. In addition, the electronic device 1000 may extract emotion information from the second voice data 603. For example, the electronic device 1000 may extract “fear” as emotion information from the second voice data 603. In this case, the electronic device 1000 may modify the default emotion information (e.g., “tension weight: 0.6, joy weight: 0.4”) based on the emotion information (e.g., fear) extracted from the second voice data 603. For example, the electronic device 1000 may modify the default emotion information (eg, “tension weight: 0.6, joy weight: 0.4”) to “tension weight: 0.6, joy weight: 0.1, fear weight: 0.3”.
[0127] The electronic device 1000 may determine the interaction type as "encouraging type (providing necessary information + adding encouraging phrases)" based on the modified emotional information (e.g., "tension weight: 0.6, joy weight: 0.1, fear weight: 0.3"). The electronic device 1000 may provide a second response 604 of an encouraging type. For example, the electronic device 1000 may output a second response 604 stating "The lecture will be at 3:00 pm (providing necessary information). Don't worry, and don't forget every material you need. Good luck. (Add encouraging phrases)".
[0128] Figure 7 is a flowchart of a method for obtaining emotion information of a user based on a progress stage of a user event, performed by the electronic device 1000 according to an embodiment.
[0129] In operation S710, the electronic device 1000 can determine the progress stage of the user event by comparing the current time with the schedule time of the user event. Here, the progress stage of the user event may include, for example, a stage before the event occurs (e.g., one day, one hour, or ten minutes), a stage during the event, or a stage after the event is completed, but is not limited thereto.
[0130] For example, when the schedule time of the user event is 3:00 pm and the current time is 2:00 pm, the electronic device 1000 may determine the progress stage of the user event as "1 hour before the user event occurs."
[0131] According to an embodiment, the current time may be the time when the speech is detected from the user. That is, the electronic device 1000 may determine whether the speech is detected from the user before the user event occurs or after the user event ends.
[0132] In operation S720, the electronic device 1000 may obtain emotion information of the user related to the user event based on the progress stage of the user event.
[0133] Because the user's emotional state to the user event may change before and after the user event, the electronic device 1000 may obtain the user's emotional information by considering the progress stage of the user event. For example, the electronic device 1000 may extract a low "stress weight" after the user event is completed compared to before the user event occurs.
[0134] In operation S730, the electronic device 1000 may determine an interaction type corresponding to the user event based on emotion information of the user.
[0135] According to an embodiment, the electronic device 1000 may determine the interaction type mapped to the emotion information based on the interaction method DB generated (or optimized) by the AI model. Operation S730 corresponds to Figure 2 Operation S220 is described below, and thus a detailed description thereof is not provided here.
[0136] In operation S740, the electronic device 1000 may provide a response to the user utterance related to the user event based on the determined interaction type.
[0137] According to an embodiment, the electronic device 1000 may extract a response mapped to an interaction type from a response DB generated (or optimized) by an AI model, and provide the extracted response to the user through an interactive interface. In this case, the response may include an active response (e.g., providing a notification of a user event). Operation S740 corresponds to Figure 2 Operation S230 is described below, and thus a detailed description thereof is not provided here.
[0138] Figure 8is a schematic diagram for describing an operation performed by the electronic device 1000 to provide a notification or a response based on a progress stage of a user event according to an embodiment.
[0139] In order to check for any new events to be registered in the event DB, the electronic device 1000 may start a conversation with the user 800 at 810 am every day by stating "Good morning. Do you have any schedule today?" In this case, when a first utterance stating "I have to give a speech to the vice president at 3:00 pm today" is detected from the user 800, the electronic device 1000 may register "giving a speech to the vice president at 3:00 pm" as a new event by analyzing the voice data of the user 800. The electronic device 1000 may obtain the emotional information corresponding to "giving a speech to the vice president" (e.g., joy weight: 0.4, tension weight: 0.6) by using an AI model, and determine the interaction type of the first utterance based on the emotional information (e.g., joy weight: 0.4, tension weight: 0.6). For example, the electronic device 1000 may determine the first encouragement type (e.g., cheer-up weight: 0.7, change topic weight: 0.3) as the interaction type. The electronic device 1000 may output a response to the first utterance based on the first encouragement type (e.g., I understand... I'll support you. Don't worry.).
[0140] Before providing a notification related to a user event to the user 800, the electronic device 1000 may compare the current time (e.g., 1:00 p.m.) with the scheduled time (e.g., 3:00 p.m.) of the user event (speaking to the vice president). The electronic device 1000 may determine, based on the comparison result, that the current time corresponds to the time 820 before the user event (e.g., speaking to the vice president) occurs.
[0141] Considering that the current time corresponds to the time 820 before the user event occurs, the electronic device 1000 can obtain the emotional information of the user 800 (e.g., joy weight: 0.1, fear weight: 0.2, tension weight: 0.7). The electronic device 1000 can determine the second encouragement type (e.g., cheer-up weight: 0.9, change topic weight: 0.1) as the interaction type corresponding to the emotional information of the user 800 (e.g., joy weight: 0.1, fear weight: 0.2, tension weight: 0.7). The electronic device 1000 can provide active interaction based on the second encouragement type. For example, the electronic device 1000 can provide a notification stating "You must be nervous about the presentation. Don't forget every material you need!" When the user 800 says "Thank you. I'll do my best..." in response to the notification of the electronic device 1000, the electronic device 1000 can output a response stating "Okay. Good luck!"
[0142] At time 830 after the user event is completed, the electronic device 1000 may query the user 800 for emotional information related to the user event. For example, the electronic device 1000 may output a query asking "How was today's presentation?" When the user 800 responds to the query and answers "Well, I made a lot of preparations, but I was still told off", the electronic device 1000 may obtain the emotional information of the user 800 by considering that the current time corresponds to time 830 after the user event is completed. In this case, the electronic device 1000 may extract emotional information from the voice data of the user 800. For example, the electronic device 1000 may obtain "regret weight: 0.4, ease weight: 0.4, joy weight: 0.2" as the emotional information of the presentation.
[0143] The electronic device 1000 may determine the empathy type (e.g., empathy weight: 0.9, providing other information weight: 0.1) as the interaction type corresponding to the emotional information of the user 800 (e.g., regret weight: 0.4, ease weight: 0.4, joy weight: 0.2). The electronic device 1000 may provide a first response stating “You have worked hard on this. What a pity” according to the determined interaction type (e.g., empathy type). When the user 800 says “Yes. I should improve myself!” in response to the first response, the electronic device 1000 may output a second response stating “You will do better next time.”
[0144] The electronic device 1000 can record the emotion information of the user 800 after the user event is completed, and use the emotion information again later. For example, when a similar event occurs to the user 800 later, the electronic device 1000 can remind the user 800 about the information related to the previous user event and prompt the user 800 to recall the good memories.
[0145] Fig. 9 is a schematic diagram for describing an operation performed by the electronic device 1000 to obtain a user's emotion information based on a person, place, or time related to a user event according to an embodiment.
[0146] According to an embodiment, in the event-emotion DB 310 connected to the AI model 300, the emotion information based on the event 900 may be subdivided based on the person, place, or time related to the event 900. In this case, the electronic device 1000 may obtain detailed emotion information of the user based on the person, place, or time related to the event 900.
[0147] For example, when the event 900 is a presentation 910, the electronic device 1000 may determine the user's emotional information as "tension weight: 0.4, joy weight: 0.6" by default. However, when information about the person to whom the presentation 910 is directed is obtained, the electronic device 1000 may determine the user's emotional information by considering the information about the person to whom the presentation 910 is directed. For example, when the person to whom the presentation 910 is directed is a chief executive officer (CEO), the electronic device 1000 may determine the user's emotional information as "tension weight: 0.9, joy weight: 0.1". Otherwise, when the person to whom the presentation 910 is directed is a friend, the electronic device 1000 may determine the user's emotional information as "tension weight: 0.1, joy weight: 0.9".
[0148] When obtaining information about the location where the presentation 910 is conducted, the electronic device 1000 may determine the user's emotion information in consideration of the location. For example, when the presentation 910 is conducted in a conference room, the electronic device 1000 may determine the user's emotion information as "tension weight: 0.4, joy weight: 0.4, fear weight: 0.2". Otherwise, when the presentation 910 is conducted during an outing, the electronic device 1000 may determine the user's emotion information as "tension weight: 0.1, joy weight: 0.9".
[0149] According to another embodiment, when the event 900 is a friend's wedding 920, the electronic device 1000 may determine the user's emotional information as "tension weight: 0.2, joy weight: 0.8" by default. However, when the time information of the friend's wedding 920 is obtained, the electronic device 1000 may determine the user's emotional information by considering the time information of the friend's wedding 920. For example, when the friend's wedding 920 is in the morning, the electronic device 1000 may determine the user's emotional information as "tension weight: 0.3, joy weight: 0.6, laziness weight: 0.1". Otherwise, when the friend's wedding 920 is in the afternoon, the electronic device 1000 may determine the user's emotional information as "tension weight: 0.2, joy weight: 0.8", or when the friend's wedding 920 is in the evening, determine the user's emotional information as "tension weight: 0.1, joy weight: 0.9".
[0150] When obtaining information about the location where the friend's wedding 920 is held, the electronic device 1000 can determine the user's emotional information in consideration of the location. For example, when the friend's wedding 920 is held outdoors, the electronic device 1000 can determine the user's emotional information as "joy weight: 1".
[0151] According to an embodiment, the electronic device 1000 may learn the user's emotional information about at least one event included in the notification information based on the user's feedback information about the at least one event. The user's feedback information is information about the user's emotional state about the at least one event, and may be obtained based on the user's input. For example, the electronic device 1000 may provide the user with notification information about at least one event through a virtual assistant interface, and obtain the user's feedback information about the at least one event included in the notification information. The electronic device 1000 may identify the user's emotional state about the at least one event by analyzing the user's feedback information, and learn the user's emotional information about the at least one event based on the user's emotional state. Now, reference will be made to Fig.10 and Fig.11 An operation performed by the electronic device 1000 to learn the user's emotion information on at least one event based on the user's feedback information is described.
[0152] Fig.10 and Fig.11 is a schematic diagram for describing an operation of personalizing the event-emotion DB 310 according to an embodiment.
[0153] Reference Fig.10 Before a user event occurs, the electronic device 1000 may provide the user with a notification related to the user event through the notification window 1010 (eg, you have one hour before your presentation).
[0154] In this case, the electronic device 1000 can provide an emotion candidate list and obtain actual emotion information from the user before the user event. For example, the electronic device 1000 can provide an emotion candidate list including happy, nervous, sad, excited, neutral, regretful, angry, annoyed, etc., and receive a user input for selecting "nervous" in the emotion candidate list.
[0155] The electronic device 1000 can personalize the default emotion information stored in the event-emotion DB 310 based on the actual emotion information input by the user. For example, the emotion information 1001 of the user before the speech stored in the event-emotion DB 310 may be "joy weight: 0.4, tension weight 0.6". However, because the user selects "tension" in the emotion candidate list displayed on the notification window 1010, the electronic device 1000 can personalize the user's emotion information for the speech by increasing the tension weight included in the default emotion information. For example, the personalized emotion information 1002 for the speech may be "joy weight: 0.1, tension weight: 0.9".
[0156] Reference Fig.11After the user event is completed, the electronic device 1000 may obtain user feedback on the user event. For example, the electronic device 1000 may provide a query window 1110 for checking emotions for the user event. According to an embodiment, the electronic device 1000 may display an emotion candidate list including a plurality of images indicating an emotional state on the query window 1110. In this case, the electronic device 1000 may receive an input from the user for selecting an image 1103 indicating joy in the emotion candidate list.
[0157] The electronic device 1000 can personalize the default emotion information for the speech stored in the event-emotion DB 310 based on the actual emotion information input by the user. For example, the user's emotion information 1101 for the speech stored in the event-emotion DB 310 can be "joy weight: 0.4, tension weight 0.6". However, because the user has selected the image 1103 indicating joy in the emotion candidate list displayed on the query window 1110, the electronic device 1000 can personalize the user's emotion information for the speech by increasing the joy weight included in the default emotion information. For example, the personalized emotion information 1102 for the speech can be "joy weight: 0.9, tension weight: 0.1".
[0158] An operation performed by the electronic device 1000 to monitor biometric information of a user by using an external device and to modify default emotion information based on the biometric information of the user will now be described.
[0159] Fig.12 is a schematic diagram for describing an operation performed by the electronic device 1000 to modify default emotion information based on biometric information according to an embodiment.
[0160] In operation S1210, the electronic device 1000 may obtain default emotion information 1201 related to the user event from the event-emotion DB 310. The default emotion information 1201 may be average emotion information of ordinary people on the user event, or previous emotion information of the user on the user event. For example, the default emotion information 1201 for the speech stored in the event-emotion DB 310 may be "joy weight: 0.4, tension weight: 0.6".
[0161] In operation S1220, in order to check the actual emotion information of the user, the electronic device 1000 may obtain the biometric information of the user from the external device 3000. For example, the electronic device 1000 may receive the heart rate information or body temperature change information of the user from a smart watch worn on the wrist of the user. In addition, the electronic device 1000 may receive the voice data of the user recorded by the smart watch. The electronic device 1000 may receive the biometric information of the user from the external device 3000 at a predetermined period or when a specific event occurs, but is not limited thereto.
[0162] According to an embodiment, the electronic device 1000 may obtain the actual emotion information of the user based on the biometric information of the user. For example, when the heart rate of the user increases and the body temperature of the user also increases, the electronic device 1000 may determine that the tension of the user increases, or when the heart rate of the user is stable, the electronic device 1000 may determine that the tension of the user decreases. In addition, when the result of analyzing the voice data of the user indicates that the voice is trembling or uses a lot of words used when nervous (e.g., good... and uh...), the electronic device 1000 may determine that the tension of the user increases. Otherwise, when the result of analyzing the voice data of the user indicates that the user cheers or laughs, the electronic device 1000 may determine that the joy of the user increases.
[0163] According to an embodiment, the electronic device 1000 may obtain the user's biometric information before, during, and after the user event registered in the event DB 340, but is not limited thereto. In this case, as the user event progresses, the electronic device 1000 may monitor changes in the user's emotions.
[0164] In operation S1230, the electronic device 1000 may modify the default emotion information 1201 for the speech stored in the event-emotion DB 310 based on the biometric information of the user. For example, when the heart rate of the user is stable, the electronic device 1000 may reduce the stress weight included in the default emotion information 1201 from "0.6" to "0.2". Therefore, the modified emotion information 1202 may be "joy weight: 0.8, stress weight: 0.2".
[0165] Fig.13 is a flowchart of a method for determining a tone based on emotion information of a user according to an embodiment.
[0166] In operation S1310, the electronic device 1000 may obtain emotion information of a user related to a user event.
[0167] According to an embodiment, the electronic device 1000 may obtain the user's emotion information related to the user event when the user speech related to the user event is detected. According to another embodiment, the electronic device 1000 may obtain the user's emotion information on the user event to actively provide interaction before the user speech related to the user event.
[0168] According to an embodiment, the electronic device 1000 can obtain the user's emotion information related to the user event by using the AI model. For example, the electronic device 1000 can obtain the user's emotion information corresponding to a specific user event from the event-emotion DB 310 generated (or optimized) by the AI model. The event-emotion DB 310 can store the average emotion information of ordinary people for a specific event, or the emotion information personalized for each user of a specific event, but is not limited thereto.
[0169] Operation S1310 may correspond to Figure 2 Operation S210 is described below, and thus a detailed description thereof is not provided here.
[0170] In operation S1320, the electronic device 1000 may determine a tone for providing a response to the user's speech based on the user's emotion information related to the user event. For example, the electronic device 1000 may determine a speech rate, a pitch average, a pitch range, an intensity, a voice quality, a pitch change, and a pronunciation based on the user's emotion information related to the user event, but is not limited thereto.
[0171] For example, refer to Fig.14 As shown in the emotion table 1410, when the user's emotion information indicates fear, the electronic device 1000 can determine the speaking speed as "much faster", the pitch average as "very high", the pitch range as "much wider", the intensity as "normal", the voice quality as "irregular phonation", the pitch change as "normal", and the pronunciation as "precise".
[0172] Otherwise, when the user's emotional information indicates anger, the electronic device 1000 may determine the speaking speed as "slightly fast", the pitch average as "very high", the pitch range as "much wider", the intensity as "higher", the voice quality as "breathy chest tone", the pitch change as "abrupt on stressed syllable", and the pronunciation as "tense".
[0173] In operation S1330, the electronic device 1000 may provide a response to the user's speech related to the user event in a determined tone. Therefore, according to an embodiment, the electronic device 1000 may effectively provide the user with emotional interaction by changing the tone of the voice used to provide the response based on the user's emotional information about the user event.
[0174] Fig.15 is a schematic diagram for describing an operation performed by the electronic device 1000 to provide a notification by using a color corresponding to emotion information of a user according to an embodiment.
[0175] According to an embodiment, the electronic device 1000 may determine a color corresponding to the user's emotional information. For example, referring to the emotional color table 1501, the electronic device 1000 may determine red for emotions such as excitement, energy, passion, action, and desire, or determine orange for emotions such as optimism, cheering up, rejuvenation, friendliness, and fun, but is not limited thereto.
[0176] According to an embodiment, the electronic device 1000 may adjust the brightness or saturation. For example, a low brightness or saturation level may be determined for an emotion close to the center of the emotion color table 1501.
[0177] According to an embodiment, the electronic device 1000 may provide a notification by using a certain color. For example, when the user event is "travel" and the user's emotional state indicates "interesting", the electronic device 1000 may display a text indicating the day before the trip on the orange notification window 1510 (e.g., you have 1 day left to travel). Otherwise, when the user event is "opening" and the user's emotional state indicates "hope", the electronic device 1000 may display a text indicating the day before the opening on the green notification window 1520 (e.g., you have 1 day left to open).
[0178] Fig.16 and Fig.17 is a schematic diagram for describing an operation of providing a diary interface performed by the electronic device 1000 according to an embodiment.
[0179] Reference Fig.16 On the first screen 1610 of the diary interface, the electronic device 1000 can display a list of user events on the diary interface. In this case, the electronic device 1000 can display the user's representative emotion for the user event in front of the event name. For example, when the user's representative emotion for his son's birthday is "happy", the electronic device 1000 can display "happy" next to his son's birthday. In addition, when the user's representative emotion for the speech is "nervous", the electronic device 1000 can display "nervous" next to the speech. According to an embodiment, the representative emotion can be displayed in the form of an emoticon.
[0180] According to an embodiment, the electronic device 1000 may sort the list of user events in chronological order or by emotion, but is not limited thereto.
[0181] Reference Fig.16In the second screen 1620, when the user selects his son's birthday in the list of user events, the electronic device 1000 can specifically provide the user with emotional information related to his son's birthday. For example, the electronic device 1000 can provide the change of the user's emotions from before the user event to after the user event in the form of a simple diary. In this case, the electronic device 1000 can display the user's emotional changes in the form of simple emoticons. In addition, when the electronic device 1000 includes a captured image related to the user event, the electronic device 1000 can add the captured image to the diary viewed by the user. The electronic device 1000 can receive additional comments related to the user event from the user. For example, the user can add an interesting plot or captured image related to his son's birthday.
[0182] Reference Fig.17 , the electronic device 1000 may provide an emotion graph 1710 corresponding to each user event on the diary interface. The emotion graph may be a graph showing the change in the user's emotions from before the event to after the event. According to an embodiment, the emotion graph may be a two-dimensional graph or a three-dimensional graph, but is not limited thereto.
[0183] Fig.18 and Fig.19 is a block diagram of an electronic device 1000 according to an embodiment.
[0184] like Fig.18 As shown, the electronic device 1000 according to the embodiment may include an outputter 1100, a processor 1300 and a memory 1700. However, not all the elements shown are essential elements. Compared with the elements shown, the electronic device 1000 may include a greater or lesser number of elements.
[0185] For example, Fig.19 As shown, the electronic device 1000 according to the embodiment may further include a sensor 1200 , a communicator 1400 , an audio / video (A / V) inputter 1500 , and a user inputter 1600 in addition to the outputter 1100 , the processor 1300 , and the memory 1700 .
[0186] The above-mentioned elements will now be described one by one.
[0187] The outputter 1100 is used to output an audio signal, a video signal, or a vibration signal, and may include, for example, a display 1111 , a sound outputter 1112 , and a vibration motor 1113 .
[0188] The sound outputter 1112 outputs audio data received from the communicator 1400 or stored in the memory 1700. In addition, the sound outputter 1112 outputs a sound signal (e.g., an incoming call sound, a message receiving sound, or a notification sound) related to the function performed by the electronic device 1000. The sound outputter 1112 may include, for example, a speaker or a buzzer. According to an embodiment, the sound outputter 1112 may provide a voice response to a user's speech related to a user event.
[0189] The vibration motor 1113 may output a vibration signal. For example, the vibration motor 1113 may output a vibration signal corresponding to the output of audio or video data (e.g., an incoming call sound or a message receiving sound). When a touch on the touch screen is input, the vibration motor 1113 may output a vibration signal.
[0190] The outputter 1100 may provide an interactive interface. The interactive interface may be provided in the form of a message or chat window or in the form of a voice input / output interface, but is not limited thereto.
[0191] The outputter 1100 may provide a diary interface including the user's emotion information related to the user event. For example, the diary interface may list the previous emotion states and episodes related to each event registered in the reminder application or calendar application. In this case, the previous emotion states may be displayed as emoticons or simple graphics.
[0192] The sensor 1200 may include at least one of a magnetic sensor 1211, an acceleration sensor 1212, a tilt sensor 1213, an infrared sensor 1214, a gyro sensor 1215, a position sensor (e.g., a global positioning system (GPS) sensor) 1216, a temperature / humidity sensor 1217, a proximity sensor 1218, and an illumination sensor 1219, but is not limited thereto. The functions of the sensors can be intuitively inferred from the names of the sensors by those of ordinary skill in the art, and thus a detailed description thereof is not provided here.
[0193] The processor 1300 generally controls the overall operation of the electronic device 1000. For example, the processor 1300 may control all of the outputter 1100, the sensor 1200, the communicator 1400, the A / V inputter 1500, the user inputter 1600, and the memory 1700 by executing a program stored in the memory 1700.
[0194] According to an embodiment, the processor 1300 may include an AI processor for generating a learning network model, but is not limited thereto. According to an embodiment, the AI processor may be implemented as a chip separate from the processor 1300.
[0195] The processor 1300 may query the user for schedule information by using the virtual assistant interface, and receive a response including information about at least one event from the user. The processor 1300 may identify the user's emotional state for at least one event by analyzing the response to the query, and learn the user's emotional information for at least one event based on the user's emotional state. The processor 1300 may determine an interaction type for at least one event based on the user's emotional information, and provide notification information for at least one event through the virtual assistant interface based on the interaction type.
[0196] The processor 1300 may obtain the user's emotion information related to the user event registered in the electronic device 1000. According to an embodiment, the electronic device 1000 may obtain the user's emotion information related to the user event from an AI model trained based on at least one of the emotion information analyzed by the expert and the user's previous emotion information. According to an embodiment, the processor 1300 may determine the progress stage of the user event by comparing the current time with the schedule time of the user event, and obtain the user's emotion information related to the user event based on the progress stage of the user event.
[0197] The processor 1300 may extract information related to the user event from the voice data of the user utterance, and register the user event as a new event in the electronic device 1000 by using the extracted information. The processor 1300 may extract emotion information of the user from the voice data of the user.
[0198] The processor 1300 may obtain default emotion information related to a user event from a table in which events are mapped to emotion information, obtain biometric information of the user by using the sensor 1200 or an external device, and obtain emotion information related to the user event by modifying the default emotion information based on the biometric information of the user. The processor 1300 may recognize an emotion state of the user for at least one event based on information obtained by analyzing voice data of a user's speech, and may learn the emotion information of the user for at least one event by modifying the default emotion information based on the emotion state of the user.
[0199] After the user event is completed, the processor 1300 can obtain feedback on the user event from the user through the user input 1600, and optimize the table in which the event is mapped to the emotion information based on the obtained feedback. For example, the processor 1300 can provide notification information of at least one event to the user through the virtual assistant interface, and obtain the user's feedback information on at least one event included in the notification information. The processor 1300 can identify the user's emotional state for at least one event by analyzing the user's feedback information, and learn the user's emotional information for at least one event based on the user's emotional state.
[0200] The processor 1300 may obtain the emotion information of the user in consideration of at least one of a person related to the user event, a schedule time of the user event, and a place related to the user event.
[0201] The processor 1300 may determine the interaction type corresponding to the user event based on the user's emotion information. According to an embodiment, the processor 1300 may determine a response to the user's speech or a tone for providing notification information based on the user's emotion information related to the user event.
[0202] The processor 1300 may control the outputter 1100 to provide a response to the user speech related to the user event through the interactive interface based on the determined interaction type. For example, the processor 1300 may select a color corresponding to the user's emotional information and display the response to the user speech or the notification information related to the user event by using the selected color.
[0203] The processor 1300 may provide a response by applying the user's preferred expression style. The processor 1300 may determine a notification providing method related to a user event in consideration of the user's condition, and provide a notification related to the user event based on the determined notification providing method. For example, when the user is in an important meeting, the processor 1300 may provide only a visual notification, and then provide a vibration or sound notification after the meeting ends.
[0204] The communicator 1400 may include one or more elements for implementing communication between the electronic device 1000 and the wearable device or between the electronic device 1000 and a server (not shown). For example, the communicator 1400 may include a short-range wireless communicator 1411, a mobile communicator 1412, and a broadcast receiver 1413.
[0205] The short-range wireless communicator 1411 may include, for example, a Bluetooth communicator, a Bluetooth low energy (BLE) communicator, a near field communicator (NFC), a wireless local area network (WLAN) (or Wi-Fi) communicator, a Zigbee communicator, an infrared data association (IrDA) communicator, a Wi-Fi direct (WFD) communicator, an ultra-wideband (UWB) communicator, and an Ant+ communicator, but is not limited thereto.
[0206] The mobile communicator 1412 transmits and receives radio signals to and from at least one of a base station in a mobile communication network, an external terminal device, and a server. Here, the radio signal may include various types of data based on transmission and reception of a voice call signal, a video call signal, or a text / multimedia message.
[0207] The broadcast receiver 1413 receives a broadcast signal and / or broadcast related information from the outside through a broadcast channel. The broadcast channel may include a satellite channel and a terrestrial channel. According to another embodiment, the electronic device 1000 may not include the broadcast receiver 1413.
[0208] According to an embodiment, the communicator 1400 may obtain biometric information of a user of an external device. For example, the communicator 1400 may collect heart rate information, breathing information, and body temperature information of the user from a wearable device connected to the electronic device 1000.
[0209] The A / V input device 1500 is used to input an audio signal or a video signal, and may include a camera 1511 and a microphone 1512. The camera 1511 may obtain an image frame (such as a still image or a moving image) through an image sensor in a video call mode or an image capture mode. The image captured by the image sensor may be processed by the processor 1300 or a separate image processor (not shown). The image frame processed by the camera 1511 may be stored in the memory 1700, or may be sent to the outside through the communicator 1400. According to an embodiment, the camera 1511 may include at least one of a telescopic camera, a wide-angle camera, and a normal camera, but is not limited thereto.
[0210] The microphone 1512 receives an external sound signal and processes it into electronic voice data. For example, the microphone 1512 may receive a sound signal from an external device or a user. The microphone 1512 may use various noise cancellation algorithms to cancel noise that occurs when receiving an external sound signal.
[0211] The user input 1600 refers to a device used by a user to input data to control the electronic device 1000. For example, the user input 1600 may include a keypad, a dome switch, a touch pad (e.g., a capacitive cover layer, a resistive cover layer, an infrared beam, a surface acoustic wave, an integral strain gauge, or a piezoelectric touch pad), a thumb wheel, or a toggle switch, but is not limited thereto.
[0212] The memory 1700 may store a program for processing and controlling the operation of the processor 1300 , and store input / output data (eg, voice data, photo images, metadata, personalized training data, and biometric information of a user).
[0213] The memory 1700 may include at least one type of storage medium including a flash memory, a hard disk, a multimedia card micro, a memory card (e.g., a secure digital (SD) or an extreme digital (XD) card), a random access memory (RAM), a static RAM (SRAM), a read-only memory (ROM), an electrically erasable programmable ROM (EEPROM), a programmable ROM (PROM), a magnetic memory, a magnetic disk, and an optical disk.
[0214] The programs stored in the memory 1700 may be classified into a plurality of modules based on their functions, for example, a response model 1711. The response model 1711 corresponds to Figure 3 The event-emotion based response model 300 is described in detail, and therefore a detailed description thereof is not provided here.
[0215] The memory 1700 may include an event DB 1712. The event DB 1712 may correspond to Figure 5 In addition, the memory 1700 may store an event-emotion DB 1713, an interaction method DB 1714, and a response DB 1715 generated by the response model 1711. The event-emotion DB 1713, the interaction method DB 1714, and the response DB 1715 correspond to Figure 3 The event-emotion DB 310, the interaction method DB 320, and the response DB 330 are described herein, and thus a detailed description thereof is not provided herein.
[0216] The electronic device 1000 may cooperate with a server (not shown) to provide a response message to the user.
[0217] In this case, the server (not shown) may generate an event-emotion based response model by learning a criterion for determining a response condition (e.g., a condition for providing an event-emotion based response), and the electronic device 1000 may determine the response condition by using the event-emotion based response model generated based on the learning result of the server (not shown). In this case, the electronic device 1000 may provide data of the event-emotion based response model to the server (not shown), and the server (not shown) may determine the response condition and provide the determined response condition to the electronic device 1000. Alternatively, the electronic device 1000 may receive data for the event-emotion based response model from the server (not shown), and determine the response condition by using the received event-emotion based response model.
[0218] The method according to the embodiment can be implemented in the form of a program command that can be executed by various computer devices, and can be recorded on a computer-readable recording medium. The computer-readable recording medium may include a program command, a data file, a data structure, or a combination thereof. The program command recorded on the computer-readable recording medium may be those program commands specially designed and constructed for the purpose of the present disclosure, or those program commands that are well known and available to those of ordinary technicians in the field of computer software. The example of a computer-readable recording medium includes a magnetic medium (e.g., a hard disk, a floppy disk, and a tape), an optical medium (e.g., a CD-ROM or a DVD), a magneto-optical medium (e.g., a floppy disk), and a hardware device (e.g., ROM, RAM, and flash memory) specially configured to store and execute program commands. The example of a program command includes a machine code generated by a compiler and a high-level language code that can be executed by a computer using an interpreter.
[0219] Some embodiments may be implemented in the form of a computer-readable recording medium, which includes instructions that can be executed by a computer, for example, a program module executed by a computer. A computer-readable recording medium may be any available medium that can be accessed by a computer, and examples thereof include all volatile, non-volatile, removable and non-removable media. A computer-readable recording medium may include computer storage media and communication media. Examples of computer storage media include all volatile, non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Examples of communication media typically include computer-readable instructions, data structures, program modules, other data or other transmission mechanisms such as modulated data signals of carrier waves, and include any information transmission media. Some embodiments may be implemented as a computer program including instructions that can be executed by a computer, for example, a computer program or computer program product executed by a computer.
[0220] Although the embodiments of the present disclosure have been described in detail, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the scope defined by the appended claims.
Claims
1. A method for providing a virtual assistant interface performed by an electronic device, the method include: By querying the user’s schedule using a virtual assistant interface; receiving a response to the query from a user, the response comprising information about at least one event related to the schedule, wherein receiving the response to the query comprises receiving voice data of an utterance of the user as the response to the query; Understanding the user's language by analyzing the response to the query to identify the at least one event related to the schedule and the user's emotional state with respect to the at least one event related to the schedule from the voice data; Based on the user's emotional state towards the at least one event related to the schedule, learning the user's emotional information towards the at least one event related to the schedule, wherein the emotional information includes at least two or more complex emotions; Determining an interaction type of at least one event related to the schedule by obtaining an interaction type corresponding to the user's emotion information on at least one event related to the schedule; Based on the interaction type, notification information of at least one event related to the schedule is provided through the virtual assistant interface.
2. The method according to claim 1, in, The learning user's emotional information includes: determining a progress stage of the at least one event based on a relationship between a current time and a scheduled time of the at least one event; and Based on the progress stage of the at least one event, the user's emotion information about the at least one event is optimized.
3. The method according to claim 1, in, Receiving a response to a query includes: extracting information related to the at least one event from the speech data; and The at least one event is registered as a new event in the electronic device by using the extracted information.
4. The method according to claim 1, in, Learning user sentiment information includes: Obtaining default emotion information related to the at least one event from information used to map events to emotion information; and Based on the user's emotional state for the at least one event related to the schedule, the user's emotional information for the at least one event is learned by modifying the default emotional information.
5. The method according to claim 1, in, Learning the user's emotion information includes learning the user's emotion information by using information about at least one of a person related to the at least one event, a schedule time of the at least one event, and a place related to the at least one event.
6. The method according to claim 1, in, Learning user sentiment information includes: After the at least one event is completed, obtaining feedback information about the at least one event from a user; and Based on the obtained feedback information, the user's emotional information about the at least one event is optimized.
7. The method according to claim 1, in, Determining the interaction type includes determining a tone for providing notification information of the at least one event based on emotion information of the user.
8. The method according to claim 1, in, The notification information provided includes: Selecting colors that correspond to the user's emotional information; and Notification information of the at least one event is displayed by using the selected color.
9. The method according to claim 1, further comprising: include: Based on the determined interaction type, a response to the user utterance related to the at least one event is provided.
10. The method according to claim 1, in, Providing notification information of the at least one event includes: determining a notification providing method related to the at least one event based on a status of the user; and Based on the determined notification providing method, notification information of the at least one event is provided.
11. The method according to claim 1, further comprising: include: A diary interface is provided, wherein the diary interface includes emotion information of the user related to the at least one event.
12. An electronic device, include: an outputter configured to provide a virtual assistant interface; as well as At least one processor configured to: By querying the user’s schedule using a virtual assistant interface; receiving a response to the query from a user, the response comprising information about at least one event related to the schedule, wherein receiving the response to the query comprises receiving voice data of an utterance of the user as the response to the query; Understanding the user's language by analyzing the response to the query to identify the at least one event related to the schedule and the user's emotional state with respect to the at least one event related to the schedule from the voice data; Based on the user's emotional state for the at least one event related to the schedule, learning the user's emotional information for the at least one event related to the schedule, wherein the emotional information includes at least two or more complex emotions; Determining an interaction type of the at least one event related to the schedule by obtaining an interaction type corresponding to the user's emotion information on the at least one event related to the schedule; and Based on the interaction type, notification information of at least one event related to the schedule is provided through the virtual assistant interface.
13. The electronic device according to claim 12, in, The at least one processor is further configured to: extracting information related to the at least one event from the speech data; and The at least one event is registered as a new event in the electronic device by using the extracted information.
14. The electronic device according to claim 12, in, The at least one processor is further configured to: obtaining default emotion information related to the at least one event from information used to map events to emotion information; as well as Based on the user's emotional state for the at least one event related to the schedule, the user's emotional information for the at least one event is learned by modifying the default emotional information.
15. A computer-readable recording medium having recorded thereon a program for executing a method, the method include: By querying the user’s schedule using a virtual assistant interface; receiving a response to the query from a user, the response comprising information about at least one event related to the schedule, wherein receiving the response to the query comprises receiving voice data of an utterance of the user as the response to the query; Understanding the user's language by analyzing the response to the query to identify at least one time related to the schedule and the user's emotional state to at least one event related to the schedule from the voice data; Based on the user's emotional state towards the at least one event related to the schedule, learning the user's emotional information towards the at least one event related to the schedule, wherein the emotional information includes at least two or more complex emotions; Determining an interaction type of at least one event related to the schedule by obtaining an interaction type corresponding to the user's emotion information on at least one event related to the schedule; Based on the interaction type, notification information of at least one event related to the schedule is provided through the virtual assistant interface.
