Electronic device and control method thereof

Abstract

The present disclosure provides an electronic device and a control method thereof. According to an embodiment of the present disclosure, the method for controlling an electronic device includes: receiving an input of a first utterance, identifying a first task for the first utterance based on the first utterance, providing a response to the first task based on a predetermined response mode, receiving an input of a second utterance, identifying a second task for the second utterance based on the second utterance, determining a correlation between the first task and the second task, and setting a response mode for the first task based on the second task based on the determined correlation satisfying a predetermined condition. The control method of the electronic device may use an artificial intelligence model trained according to at least one of a machine learning, a neural network, or a deep learning algorithm.

Description

Technical Field

[0001] The present disclosure relates to an electronic device and a control method thereof, and, for example, to an electronic device capable of providing a response to a user utterance and an additional response related to the user utterance and a control method thereof.

[0002] The present disclosure also relates to artificial intelligence (AI) systems and their applications, such as using machine learning algorithms to simulate functions of the human brain, such as cognition and determination. Background Art

[0003] Artificial intelligence (AI) systems may refer to computer systems that achieve human-level intelligence, and may include systems in which machines self-learn, determine, and become more intelligent, unlike conventional rule-based intelligent systems. Artificial intelligence systems show more improved recognition rates as they are used more, and become more able to correctly understand user preferences. For this reason, conventional rule-based intelligent systems are gradually being replaced by deep learning-based artificial intelligence systems.

[0004] Artificial intelligence technology may include machine learning (deep learning) and element technology that utilizes machine learning.

[0005] Machine learning may refer to, for example, an algorithmic technology that classifies / learns features of input data by itself. Meanwhile, element technology may refer to, for example, a technology that utilizes a machine learning algorithm such as deep learning, and includes technical fields such as language understanding, visual understanding, reasoning / prediction, knowledge representation, and operation control.

[0006] Example applications of various fields of application of artificial intelligence technology are as follows. Language understanding may refer to, for example, the technology of recognizing human language / characters and applying / processing them, and includes natural speech processing, machine translation, dialogue system, query and answer, speech recognition / synthesis, etc. Visual understanding may refer to, for example, the technology of recognizing objects and processing objects in a manner similar to human vision, and may include object recognition, object tracking, image search, human recognition, scene understanding, spatial understanding, image improvement, etc. Inference / prediction may refer to, for example, the technology of determining information and making logical inferences and predictions, and may include knowledge / probability-based inference, optimization prediction, preference-based planning, recommendation, etc. Knowledge representation may refer to, for example, the technology of automatically processing information of human experience into knowledge data, and may include knowledge construction (data generation / classification), knowledge management (data utilization), etc. Operation control may refer to, for example, the technology of controlling the autonomous driving of vehicles and the movement of robots, and may include motion control (navigation, collision, driving), operation control (behavior control), etc.

[0007] Meanwhile, in the case of an electronic device including a dialogue system that provides a response to a user's query, the electronic device only provides a response to the user's query, but does not provide an additional response to the user's query. Therefore, it is inconvenient that the user must perform a large number of queries or long queries to obtain desired information, and there is also a problem that the dialogue time between the user and the dialogue system becomes longer. Summary of the invention

[0008] Technical issues

[0009] Embodiments of the present disclosure solve the above-mentioned problems and relate to an electronic device and a control method thereof that can provide a response to a user utterance and an additional response related to the user utterance based on the user's preference information and a conversation history.

[0010] Technical Solution

[0011] An exemplary control method for an electronic device according to an exemplary embodiment includes the following steps: receiving an input of a first utterance; identifying a first task for the first utterance based on the first utterance; providing a response to the first task based on a predetermined response mode; receiving an input of a second utterance; identifying a second task for the second utterance based on the second utterance; determining an association between the first task and the second task; and setting a response mode for the first task based on the second task based on the determined association satisfying a predetermined condition.

[0012] The response pattern may be determined while including at least one of information related to the length of the response to the utterance or information related to the type of information included in the response to the utterance.

[0013] The predetermined response mode may be a response mode selected by a command or a response mode automatically set based on a utterance history of the electronic device.

[0014] The method may output the speech according to the first utterance to a trained artificial intelligence model to obtain information about the acoustic features of the first utterance, and identify the user based on the information about the obtained acoustic features. A predetermined response mode may be determined based on the identified conversation history and the user's preference information.

[0015] The predetermined condition may be a condition that information related to the degree of association between the first task and the second task is equal to or greater than a threshold value.

[0016] The control method may further include the steps of: determining whether a third task for the third utterance is associated with the first task based on receiving an input of the third utterance; and providing a response based on the set response mode for the first task.

[0017] The control method may further include the steps of receiving an input of a third utterance, and identifying a third task for the third utterance based on the third utterance, and determining a degree of association between the first task and the third task. The setting step may further include the steps of determining priorities of the second task and the third task based on the determined degree of association between the first task and the third task satisfying a predetermined condition, and setting a response mode for the first task based on the determined priorities, the second task, and the third task.

[0018] The control method may include the steps of acquiring feedback for the provided response, and updating the degree of association between the first task and the second task based on the acquired feedback.

[0019] The control method may further include the following steps: based on the degree of association between the first task and the second task, storing the first task and the second task in the form of an ontology.

[0020] At least one of the steps of identifying the first task, determining the relevance, setting the response mode, or providing the response may be performed by an artificial intelligence model. The artificial intelligence model may include a plurality of layers each including at least one node, and each of the at least one node may include a neural network model having connection weights for interpreting input values.

[0021] An exemplary electronic device according to an exemplary embodiment of the present disclosure may include a memory configured to store at least one command and a processor configured to execute at least one command to control the electronic device. The processor may control the electronic device to: receive an input of a first utterance, identify a first task for the first utterance based on the first utterance, provide a response to the first task based on a predetermined response mode, receive an input of a second utterance, identify a second task for the second utterance based on the second utterance, determine a correlation between the first task and the second task, and set a response mode for the first task based on the second task based on the determined correlation satisfying a predetermined condition.

[0022] The response pattern may be determined while including at least one of information related to the length of the response to the utterance or information related to the type of information included in the response to the utterance.

[0023] The predetermined response mode may be a response mode selected by a command or a response mode automatically set based on a utterance history of the electronic device.

[0024] The predetermined condition may be a condition that information related to the degree of association between the first task and the second task is equal to or greater than a threshold value.

[0025] The processor can control the electronic device to input the speech according to the first utterance into the trained artificial intelligence model to obtain information about the acoustic features of the first utterance, and identify the user based on the information about the obtained acoustic features. The predetermined response mode is determined based on the identified conversation history and the user's preference information.

[0026] The processor may control the electronic device to determine whether a third task for the third utterance is associated with the first task based on receiving an input of the third utterance, and provide a response based on the set response mode for the first task.

[0027] The processor can control the electronic device to receive input of a third sound, identify a third task for the third sound based on the third sound, determine the association between the first task and the third task, and determine the priorities of the second task and the third task based on the determined association between the first task and the third task satisfying a predetermined condition, and set a response mode for the first task based on the determined priorities of the second task and the third task.

[0028] Furthermore, the processor may control the electronic device to obtain feedback for the provided response, and update the degree of association between the first task and the second task based on the obtained feedback.

[0029] The processor may control the electronic device to store the first task and the second task in the memory in the form of a body based on the degree of association between the first task and the second task.

[0030] At least one of identifying the first task, determining the relevance, setting the response mode, or providing the response can be performed by an artificial intelligence model. The artificial intelligence model may include a plurality of layers each including at least one node, and each of the at least one node may include a neural network model having connection weights for interpreting input values.

[0031] Beneficial Effects

[0032] According to the above-described various exemplary embodiments of the present disclosure, the electronic device may provide various responses to the user's utterance and thereby reduce unnecessary conversation turns. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The above and other aspects, features, and advantages of certain embodiments of the present disclosure will become more apparent from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0034] Figure 1 is a diagram illustrating an exemplary operation of an exemplary electronic device according to an embodiment of the present disclosure;

[0035] Figure 2is a block diagram illustrating an exemplary configuration of an exemplary electronic device according to an embodiment of the present disclosure;

[0036] Figure 3 is a block diagram illustrating an exemplary configuration of an exemplary electronic device according to an embodiment of the present disclosure;

[0037] Figure 4 is a block diagram illustrating an exemplary dialog system according to an embodiment of the present disclosure;

[0038] Figure 5a is a flow chart illustrating an exemplary method for providing a response to a user utterance according to an embodiment of the present disclosure;

[0039] Figure 5b is a diagram illustrating an exemplary method for providing an additional response according to another embodiment of the present disclosure;

[0040] Figure 6a is a diagram illustrating an exemplary method for updating information related to the degree of association between tasks according to an embodiment of the present disclosure;

[0041] Figure 6b is a diagram showing an example of information related to the degree of association between tasks stored in the knowledge database 460 according to an embodiment of the present disclosure;

[0042] Figure 7 is a diagram showing an example of providing a response to a user utterance according to an embodiment of the present disclosure;

[0043] Figure 8a is a diagram showing an example in which a response to a user utterance is of a query type according to another embodiment of the present disclosure;

[0044] Figure 8b is a diagram showing an example in which a response to a user utterance is of a query type according to another embodiment of the present disclosure;

[0045] Fig. 9 is a flow chart illustrating an exemplary method of providing a query-type response to a user utterance according to an embodiment of the present disclosure;

[0046] Fig.10 is a diagram showing an example of providing a response to a user's utterance according to yet another embodiment of the present disclosure;

[0047] Fig.11 is a diagram showing an exemplary system including an electronic device and a server according to an embodiment of the present disclosure;

[0048] Fig.12 is a flowchart illustrating an exemplary method of controlling an electronic device according to an embodiment of the present disclosure; and

[0049] Fig.13 is a flowchart illustrating an exemplary method of controlling an electronic device according to another embodiment of the present disclosure. DETAILED DESCRIPTION

[0050] Hereinafter, various exemplary embodiments of the present disclosure will be described with reference to the accompanying drawings. At the same time, it should be noted that the various exemplary embodiments are not intended to limit the technology described in the present disclosure to a specific embodiment, but should be interpreted as including various modifications, equivalents and / or substitutions of the embodiments of the present disclosure. In addition, with respect to the description of the accompanying drawings, similar reference numerals may be used for similar components.

[0051] In the present disclosure, expressions such as “having”, “may have”, “including” and “may include” should be understood to indicate the existence of such features (for example, elements such as values, functions, operations and components), and these expressions are not intended to exclude the existence of additional features.

[0052] In the present disclosure, the expression "A or B", "at least one of A and / or B", or "one or more of A and / or B", etc. may include all possible combinations of the listed items. For example, "A or B", "at least one of A and B", or "at least one of A or B" refers to all of the following situations: (1) including at least one A, (2) including at least one B, or (3) including at least one A and at least one B.

[0053] The expressions "first", "second", etc. used in the present disclosure may be used to describe various elements, regardless of any order and / or importance. Such expressions may be used to distinguish one element from another element, but are not intended to limit these elements.

[0054] In the present disclosure, the description that one element (e.g., a first element) is "(operably or communicatively) coupled to" or "connected to" another element (e.g., a second element) should be understood to include the case where one element is directly coupled to another element, and the case where one element is coupled to another element via another element (e.g., a third element). The description that one element (e.g., a first element) is "directly coupled" or "directly connected" to another element (e.g., a second element) may be interpreted to mean the case where there is no another element (e.g., a third element) between one element and another element.

[0055] The expression "configured to" used in the present disclosure may be used interchangeably with other expressions, such as "suitable for", "capable of", "designed to", "suitable for", "made of", and "capable of", as appropriate. The term "configured to" does not necessarily refer to a device that is "specially designed to" in terms of hardware. On the contrary, in some cases, the expression "a device configured to" may refer to a device that is "capable of" performing an operation together with another device or element. For example, the phrase "a sub-processor configured to perform A, B, and C" may refer to, for example, but not limited to, a dedicated processor (e.g., an embedded processor) for performing the corresponding operations, a general-purpose processor (e.g., a CPU or an application processor) that can perform the corresponding operations by executing one or more software programs stored in a storage device, and the like.

[0056] The electronic device according to various exemplary embodiments of the present disclosure may include, for example, but not limited to, at least one of a smart phone, a tablet PC, a mobile phone, a video phone, an e-book reader, a desktop PC, a laptop, a netbook, a workstation, a server, a PDA, a portable multimedia player (PMP), an MP3 player, a medical device, a camera, a wearable device, etc. The wearable device may include, for example, but not limited to, at least one of an accessory device (e.g., a watch, a ring, a bracelet, an anklet, a necklace, glasses, contact lenses, or a head mounted device (HMD)), a device integrated with fabric or clothing (e.g., electronic clothing), a body attachment device (e.g., a skin pad or a tattoo), an implantable circuit, etc. In some embodiments, the electronic device may include, for example, but not limited to, at least one of a television, a digital video disc (DVD) player, an audio, a refrigerator, an air conditioner, a cleaner, an oven, a microwave, a washing machine, an air purifier, a set-top box, a home automation control panel, a security control panel, a media box (e.g., Samsung HomeSyncTM, Apple TVTM, or GoogleTVTM), a game console (e.g., XboxTM, PlayStationTM), an electronic dictionary, an electronic key, a camcorder, an electronic photo frame, and the like.

[0057] In another exemplary embodiment of the present disclosure, the electronic device may include, for example, but not limited to, at least one of various types of medical instruments (for example, various types of portable medical measuring instruments (glucose meter, heart rate meter, blood pressure meter or thermometer, etc.), magnetic resonance angiography (MRA), magnetic resonance imaging (MRI), computed tomography (CT), photographing device or ultrasonic instrument, etc.), navigation equipment, global navigation satellite system (GNSS), event data recorder (EDR), flight data recorder (FDR), vehicle infotainment equipment, electronic devices for ships (for example, navigation equipment for ships, gyroscopes, etc.), avionics devices, security equipment, head units (control units) for vehicles, industrial or household robots, drones, ATMs of financial institutions, points of sale (POS) of stores, Internet of Things (IoT) devices (for example, light bulbs, various types of sensors, sprinkler devices, fire alarms, thermostats, street lights, ovens, exercise equipment, hot water tanks, heaters, boilers, etc.).

[0058] In the present disclosure, the term "user" may refer to a person using an electronic device or an apparatus (eg, an artificial intelligence electronic device) using an electronic device.

[0059] Hereinafter, the present disclosure will be described in more detail with reference to the accompanying drawings.

[0060] Figure 1 is a diagram illustrating an exemplary operation of an electronic device according to an embodiment of the present disclosure.

[0061] like Figure 1 As shown on the left side of FIG. 1 , the electronic device 100 (eg, referring to FIG. 1 ) Figure 2 ) may include a dialog system for providing responses to user queries. Conventional dialog systems can only provide responses to user queries, such as Figure 1 For example, the electronic device 100 may only provide a response about the temperature to a query about the temperature, and only provide a response about the weather to a query about the weather, and only provide a response about the degree of air pollution to a query about the degree of air pollution.

[0062] according to Figure 1 In the traditional dialogue system shown on the left side of , the user can perform corresponding queries about temperature checks, weather checks, and air pollution checks and obtain responses to the corresponding queries, or perform queries about all temperature checks, weather checks, and air pollution checks and obtain responses.

[0063] However, even when the user of the electronic device 100 mainly performs inquiries about the weather, inquiries about the degree of atmospheric pollution, and inquiries about the temperature, the electronic device 100 only provides responses to the user's inquiries, and thus generates unnecessary conversation turns. Therefore, according to an embodiment of the present disclosure, in order to reduce unnecessary conversation turns, the electronic device 100 may learn the user's conversation history, and Figure 1 As shown on the right side of , a query asking about the weather provides information about temperature and air pollution.

[0064] For example, the electronic device 100 may provide responses to user queries and responses to additional queries that the user expects based on, for example, the user's conversation history, preference information set by the user, and the like.

[0065] According to an embodiment of the present disclosure, the electronic device 100 may receive an input of a first utterance of a user, identify a first task for the first utterance, and provide a response to the first task according to a predetermined response mode. Thereafter, the electronic device 100 may receive an input of a second utterance, identify a second task for the second utterance, and set a response mode for the first task based on the second task based on the association between the first task and the second task satisfying a predetermined condition. Therefore, in the case where the first utterance is input again later, a response may be provided based on the second task according to the response mode for the first task set.

[0066] The response mode may refer to, for example, at least one of the length of the response, the type of information included in the response, and / or the number of information pieces included in the response. The setting of the response mode may include newly generating a response mode and changing a previous response mode. The setting of the response mode may be performed by a user through a manual method, or may be performed automatically.

[0067] According to an embodiment, the electronic device 100 may perform acquisition or recognition of a user. Therefore, the electronic device 100 may store different user conversation histories and preference information set by the user separately for each user.

[0068] For example, when the electronic device 100 logs in to an AI-based service, such as a smart phone, the electronic device 100 may update the conversation history and preference information of the currently logged in user based on the user's utterance.

[0069] When the electronic device 100 is a public device (such as an AI speaker) that can be used by multiple users, the electronic device 100 can identify the user based on the user's voice. The process of identifying the user may include, for example, the process of determining the user based on the characteristics of the user's voice, such as the pitch, intonation, pronunciation, and speed. For example, the electronic device 100 may input the received user's voice into a trained artificial intelligence model, and obtain information about the acoustic features of the user's voice through the trained artificial intelligence model, such as information about the voice of the user's voice, information about the unit phoneme of the user's voice, etc. In addition, the electronic device 100 may compare the obtained information about the acoustic features of the user's voice with the pre-stored information about the acoustic features of the user's voice, and identify the user who is the subject of the received user voice.

[0070] Based on recognizing the user according to the user's utterance, the electronic device 100 may update the recognized conversation history and the user's preference information.

[0071] When the electronic device 100 includes a camera, the electronic device may obtain a user image through the camera and perform object recognition based on the obtained user image to recognize the user.

[0072] For example, the electronic device 100 may extract features of an object included in the obtained user image, obtain a probability that the object included in the obtained user image corresponds to each of a plurality of categories for classifying objects based on the extracted object features, and obtain the object included in the obtained user image. When the object included in the obtained user image is recognized, the electronic device 100 may compare information about the recognized object with information about the object included in the pre-stored user image, and recognize the user corresponding to the user image.

[0073] Like the process of identifying a user based on the user's voice, the process of identifying a user based on the user's image can be performed, for example, by an artificial intelligence model, which includes an artificial neural network, such as but not limited to a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a generative adversary network (GAN), etc.

[0074] Figure 2 is a block diagram showing an exemplary configuration of an exemplary electronic device according to an embodiment of the present disclosure. Figure 2 As shown, electronic device 100 may include memory 110 and processor 120 (eg, including processing circuitry).

[0075] The memory 110 may store commands or data related to at least one other component of the electronic device 100. For example, the memory 110 may be implemented as a non-volatile memory, a volatile memory, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), etc. In addition, the processor 120 may access the memory 110, and the processor 120 performs reading / recording / correction / deletion / update of data, etc. In the present disclosure, the term memory may include the memory 110, a ROM (not shown) within the processor 120, a RAM (not shown), or a memory card (not shown) (e.g., a micro SD card, a memory stick) installed on the electronic device 100. In addition, in the memory 110, programs and data for constituting various types of screens to be displayed in the display area of ​​the display 150, etc. may be stored.

[0076] The memory 110 may store a dialog system that provides responses to user input (eg, user utterances). Figure 4 As shown, the dialog system may include, for example, but not limited to, an automatic speech recognition (ASR) part (e.g., including a processing circuit and / or an executable program element) 410, a natural language understanding (NLU) part (e.g., including a processing circuit and / or an executable program element) 420, a dialog manager (DM) (e.g., including a processing circuit and / or an executable program element) 430. A natural language generation (NLG) part (e.g., including a processing circuit and / or an executable program element) 440, a text to speech (TTS) (e.g., including a processing circuit and / or an executable program element) 450, and a knowledge database 460.

[0077] The automatic speech recognition part 410 may include various processing circuits and / or executable program elements for converting the user's speech into a form of text that can be processed by the electronic device 100 by performing speech recognition on the user's speech input via a microphone. The automatic speech recognition part 410 may include a language model for correcting conversion errors, the user's unique speech, speech errors, etc. The natural language understanding part 420 may include various processing circuits and / or executable program elements for identifying tasks related to the intent of the entity and the user's speech based on the result of speech recognition. For example, the natural language understanding part 420 may perform sentence analysis by analyzing the structure and main components of the sentence and interpreting the sentence using statistics / analysis, etc. The dialogue manager 430 may include various processing circuits and / or executable program elements for obtaining information about the response of the user's speech based on the result of natural language understanding and the data stored in the knowledge database 460. The dialogue manager 430 may be implemented, for example, based on frames, based on agents, etc., and may be implemented by modeling based on Markov decision processes (MDPs) and reinforcement learning. The dialogue manager 430 may acquire information for generating a response, and as described above, the acquired information may be determined based on the task identified by the natural language understanding portion 420 and the data stored in the knowledge database 460. The natural language generation portion 440 may include various processing circuits and / or executable program elements, and may acquire natural language as a response to the user's utterance based on the information acquired by the dialogue manager 430. The TTS 450 may include various processing circuits and / or executable program elements, and convert the acquired natural language into speech. In the above manner, the dialogue system may provide a response to the user's utterance as speech, and the user may perform a dialogue with the electronic device 100.

[0078] For example, the natural language generation part 440 according to an embodiment of the present disclosure may input information acquired through the dialogue manager 430 and the knowledge database 460 as input values ​​of the artificial intelligence model, and acquire natural language as a response to the user's utterance.

[0079] The knowledge database 460 may, for example, store information necessary for generating a response at the dialog manager 430. The data stored in the knowledge database 460 may be different. For example, the knowledge database 460 may store user preference information. The user preference information may relate to, for example, a user-preferred response type regarding a user utterance. The response type that the user likes may, for example, be a response type that provides a response to the user utterance or a response type that provides an additional query regarding the user utterance. The response type that the user prefers may include information for setting details of the response (or query) for each of the response types that provide a response to the user utterance or the response type that provides an additional query regarding the user utterance. For example, the user preference information may be a type that provides an additional query regarding the user utterance, and the additional query may be information regarding a query constituted in the form of "yes" or "no". The user preference information may, for example, be a type that provides an additional query regarding the user utterance, and the additional query may be information regarding a query constituted to select one of a plurality of alternatives. The user preference information may be information regarding a type that provides a response to the user utterance and information regarding a plurality of pieces of information included in the response.

[0080] The knowledge database 460 may store the user's past utterance history. For example, the knowledge database 460 may store historical information related to past user utterances and responses to the utterances, and re-utterances related to the responses. The method of storing historical information may be different. For example, the historical information may include an ontology that includes information about the degree of association between multiple tasks, and the multiple tasks are related to multiple user utterances. The historical information may be in a form in which information about the user utterance, the response to the utterance, and information about the re-utterance of the response are stored in the form of a data set (e.g., (query, response)).

[0081] The memory 110 may store an artificial intelligence agent for operating the dialogue system. For example, the electronic device 100 may use the artificial intelligence agent to generate natural language as a response to the user's utterance. The artificial intelligence agent may refer to, for example, a dedicated program for providing artificial intelligence (AI)-based services (e.g., speech recognition services, agent services, translation services, search services, etc.), and may be, but is not limited to, executed by, for example, a conventional general-purpose processor (e.g., a CPU) or a separate AI-specific processor (e.g., a GPU, etc.).

[0082] For example, the AI ​​agent can operate in the case of inputting user utterances. The AI ​​agent can input user queries into a trained AI learning model and obtain responses. If the user utterance is input (e.g., a trigger voice for executing an AI function)

[0083] Or select a predetermined button (for example, a button for executing an artificial intelligence agent function), the artificial intelligence agent can operate. The artificial intelligence agent may have been executed before the user utterance is input or the predetermined button is selected. In this example, after the user utterance is input or the predetermined button is selected, the artificial intelligence agent of the electronic device 100 can obtain natural language as a response to the user utterance. The artificial intelligence agent may be in a standby state before the user utterance is input or the predetermined button is selected. The standby state may refer to a state in which, for example, receipt of a predetermined user input is detected for controlling the start of the operation of the artificial intelligence agent. If a user utterance is input or a predetermined button is selected when the artificial intelligence agent is in the standby state, the electronic device 100 can operate the artificial intelligence agent and obtain natural language as a response to the user utterance.

[0084] According to an embodiment of the present disclosure, the memory 110 may store an artificial intelligence model trained to generate (or obtain) natural language. In the present disclosure, fields such as the computer performance of the recognition model application or the device may be considered to construct a trained artificial intelligence model. For example, the artificial intelligence model may be trained to obtain natural language using information obtained from the dialogue manager 430 and the knowledge database 460 as input data. In order to generate natural natural language, the trained artificial intelligence model may be a neural network-based model. The artificial intelligence model may be designed, for example, to simulate the structure of the human brain on a computer, and may include a plurality of network nodes having weights of neurons simulating a human neural network. A plurality of network nodes may each form a connection relationship to simulate the synaptic activity of neurons exchanging signals via synapses. In addition, the document summary model may include, for example, a neural network model, or a deep learning model developed from a neural network model. In a deep learning model, a plurality of network nodes may be located at different depths (or layers) from each other, and data may be exchanged according to the relationship of convolutional connections. Examples of trained artificial intelligence models may include, for example, but not limited to, a deep neural network (DNN), a recurrent neural network (RNN), a bidirectional recurrent deep neural network (BRDNN), etc., but the trained artificial intelligence models are not limited thereto.

[0085] In the above embodiment, it is described that the artificial intelligence model is stored in the electronic device 100, but this is only an example, and the artificial intelligence model may be stored in another electronic device. For example, the artificial intelligence model may be stored in at least one external server. The electronic device 100 can receive the input of the user's voice and send the voice to the external server storing the artificial intelligence model, and the artificial intelligence model stored in the external server can input the user's voice received from the electronic device 100 as an input value and output a result.

[0086] The processor 120 may include various processing circuits, and is electrically connected to the memory 110 , and controls overall operations and functions of the electronic device 100 .

[0087] For example, the processor 120 may receive an input of a user utterance, and determine a task related to the user utterance based on the input user utterance. For example, the task related to the user utterance may include information necessary for a response to the user utterance. For example, in the case where the user utterance is a query to inquire about the weather, the task may be a weather check. In the example where the user utterance is a query to inquire about the temperature, the task may be a temperature check. In the example where the user utterance is a query to inquire about the degree of atmospheric pollution, the task may be an atmospheric pollution check.

[0088] The processor 120 may determine the response mode of the task related to the user utterance and the additional task related to the task based on the task related to the user utterance. For example, the processor 120 may identify a first task for the first user utterance and a second task for the second user utterance. If the association between the first task and the second task meets a predetermined condition, the processor 120 may set a response mode for the first utterance based on the second task. For example, in the case where the first task is associated with the second task, the processor 120 may determine the response mode based on the first task and the second task. The predetermined condition may, for example, be a condition that information about the association between the first task and the second task is equal to or greater than a threshold.

[0089] The response mode may be determined by various methods. For example, the response mode may include at least one of information related to the length of the response to the user utterance or information related to the type of information included in the response to the user utterance, and may be determined based on the above information. The response mode may be determined based on an utterance history and user preference information, wherein the utterance history includes user utterances and additional utterances for performing additional tasks. The additional task (or second task) may be a task related to the determined task (or first task). For example, in the case where the determined task is a temperature check, the additional task may be a weather check, an air pollution check, etc., which are highly correlated with the temperature check.

[0090] The response mode may be a response mode selected by a user command, or a response mode automatically generated based on a utterance history. For example, in the case where there is a user command having a condition of inputting a first utterance and a second utterance within a predetermined time period, if the second utterance is input within a predetermined time after the first utterance is input, the processor 120 may determine that a first task for the first utterance and a second task for the second utterance are associated with each other, and determine a response mode. If a user command for utterances for a predetermined category is input, the processor 120 may determine that a first task for the first utterance and a second task for the second utterance are associated with each other with respect to the category, and determine a response mode. In the case where a user command is associated with the length and type of a response, the processor 120 may determine a response mode based on the length of the response or the type of response determined according to the user command. The processor 120 may determine a response mode according to the various user commands described above, but the present disclosure is not limited thereto, and the response mode may be automatically set.

[0091] The processor 120 may acquire response information for tasks related to the user's utterance and response information for additional tasks, and provide the acquired response information and additional response information to the user. For example, the processor 120 may provide a response for a determined response mode based on a first task and a second task.

[0092] The processor 120 may store information about the degree of association between the task and the additional task in the memory 110. For example, the processor 120 may store information about the degree of association between the task and the additional task in the knowledge database 460.

[0093] The additional task (or second task) may be a task that is associated with the task determined according to the user utterance to a degree greater than a threshold. For example, the processor 120 may determine a task whose information about the degree of association with the task determined according to the user utterance is greater than the threshold as an additional task. For example, when the task determined according to the user utterance is temperature check, and the degree of association between the temperature check task and the weather check task is 0.9, and the degree of association between the temperature check task and the air pollution check task is 0.5, and the threshold is 0.7, the processor 120 may determine the weather check task whose degree of association with the temperature check task is greater than 0.7 as an additional task, and may not determine the air pollution check task whose degree of association with the temperature check task is less than 0.7 as an additional task.

[0094] The processor 120 may receive an input of a third utterance of the user, identify a third task for the third utterance, and determine a degree of association between the identified third task and the first task. If the degree of association between the first task and the third task satisfies a predetermined condition, the processor 120 may determine a priority of the second task and the third task, and set a response mode for the first utterance based on the determined priority, the second task, and the third task.

[0095] For example, in the case of determining multiple additional tasks, the processor 120 may determine information related to the degree of association between the task for the user's utterance and the multiple determined tasks, and determine the order of providing responses to the additional tasks based on the determined information related to the degree of association. For example, when the task for the user's utterance is a temperature check task, and the determined additional tasks are a weather check task, an air pollution check task, and a population density check, and the degree of association between the temperature check task and the weather check task is 0.9, and the degree of association between the temperature check task and the air pollution check task is 0.5, and the degree of association between the temperature check task and the population density check task is 0.3, the processor 120 may first provide additional responses for tasks with high associations. For example, regarding the user's utterance "What is the temperature in Seoul today?", the processor 120 may provide responses in the order of the response of the temperature check task uttered by the user, the additional response of the weather check task, the additional response of the air pollution check task, and the additional response of the population density check task, such as "The temperature in Seoul today is 20 degrees, the weather is good, the concentration of fine dust is 10, and the population density is low".

[0096] According to another embodiment of the present disclosure, for a response mode set for a first task for a first utterance based on a second task associated with the first task, if a third utterance different from the first and second utterances is input, the processor 120 may determine whether the third task for the third utterance is associated with the first task for the first utterance. If there is an association, the processor 120 may provide a response according to the response mode set for the first task.

[0097] The processor 120 may obtain feedback of the response provided, and update the association between the first task and the second task based on the obtained feedback. For example, the processor 120 may obtain user feedback on the response information and the additional response information provided for the user utterance, and update information related to the association between the task used for the user utterance and the additional task based on the obtained feedback. For example, regarding the user utterance "What is the temperature in Seoul today?", a response such as "The temperature in Seoul today is 20 degrees, the weather is good, the fine dust concentration is 10, and the population density is low" is provided. If the user feedback end response is input after the weather information, the processor 120 may reduce the association between the temperature check task and the air pollution check task and the association between the temperature check task and the population density check task.

[0098] The processor 120 may receive input regarding response information provided for a user utterance and an additional user utterance to which the additional response information is provided, and update information regarding the degree of association between the task for the user utterance and the task for the additional user utterance based on the task for which the additional user utterance is input. For example, if a response is provided regarding a user utterance "What is the temperature in Seoul today?", such as "The temperature in Seoul today is 20 degrees, and the weather is good", and then an additional user utterance is provided, such as "What is the fine dust concentration in Seoul today?" as input, the processor 120 may provide a response, such as "The fine dust concentration in Seoul today is good, at 10", and update information regarding the degree of association between the temperature check task and the air pollution check task.

[0099] The processor 120 may provide response information to the user's utterance and provide a query message for providing additional response information. That is, the processor 120 may not provide response information and additional response information to the user's utterance at the same time, but may provide a query message to inquire whether to provide additional response information. For example, if the user utters "What is the temperature in Seoul today?", the processor 120 may provide a response to the user's utterance and a query message, such as "The temperature in Seoul today is 20 degrees. Do you want to know the weather in Seoul?". If a user command for a query message is received, the processor 120 may provide an additional response to the user command. For example, the processor 120 may provide a response to the user's utterance and a query message for providing additional response information, such as "Do you want to know the weather in Seoul?". If the processor 120 receives a user command "Yes" to the user query "Do you still want to know the weather in Seoul?", the processor 120 may provide an additional response, such as "The weather in Seoul is good today".

[0100] The processor 120 may store the task for user utterance and the additional tasks related to the task for user utterance in the memory 110 in the form of, for example, an ontology based on the information about the degree of association between the task for user utterance and the additional tasks related to the task for user utterance. For example, the processor 120 may store the first task and the second task in the memory 110 in the form of an ontology based on the degree of association between the first task and the second task.

[0101] The above-mentioned various operations of the processor 120 may be performed by an artificial intelligence model. For example, at least one step of the operation of determining a task related to the user's utterance, the operation of determining a response mode, the operation of obtaining additional response information, and the operation of providing additional response information is performed by the artificial intelligence model, and the artificial intelligence model may include a plurality of layers each including at least one node, and each of the at least one node may be a neural network model having a connection weight for interpreting an input value.

[0102] Figure 3 is a block diagram illustrating an exemplary configuration of an exemplary electronic device according to an embodiment of the present disclosure.

[0103] like Figure 3 As shown, in addition to the memory 110 and the processor 120, the electronic device 100 may further include a communicator (e.g., including a communication circuit) 130, an inputter (e.g., including an input circuit) 140, a display 150, and an audio outputter (e.g., including an audio output circuit) 160. However, the components of the electronic device 100 are not limited to the above components, and some components may be added or omitted as needed.

[0104] The communicator 130 may include various communication circuits and communications with external devices. The communication connection between the communicator 130 and the external device may include communication through a third device (e.g., a repeater, a hub, an access point, a server, or a gateway). Wireless communication may include, for example, but not limited to, cellular communication using at least one of LTE, LTE Advanced (LTE-A), Code Division Multiple Access (CDMA), Wideband CDMA (WCDMA), Universal Mobile Telecommunications System (UMTS), Wireless Broadband (WiBro), Global System for Mobile Communications (GSM), etc. According to an embodiment, wireless communication may include, for example, wireless fidelity (WiFi), Bluetooth, Bluetooth Low Energy (BLE), Zigbee, near field communication (NFC), magnetic security transmission, radio frequency (RF), or body area network (BAN). Wired communication may include, for example, at least one of Universal Serial Bus (USB), High Definition Multimedia Interface (HDMI), Recommended Standard 232 (RS-232), Power Line Communication, or Plain Old Telephone Service (POTS). The network performing wireless communication or wired communication may include a telecommunication network such as at least one of a computer network (eg, LAN or WAN), the Internet, or a telephone network.

[0105] The inputter 140 may include various input circuits and receive an input of a user command. The inputter 140 may include, for example but not limited to, a camera 141, a microphone 142, a touch panel 143, and the like.

[0106] The camera 141 may include various circuits for acquiring image data around the electronic device 100. The camera 141 may capture still images and moving images. For example, the camera 141 may include one or more of an image sensor (e.g., a front surface sensor or a rear surface sensor), a lens, an image signal processor (ISP), or a flash memory (e.g., an LED or a xenon lamp, etc.). The microphone 142 may include various circuits for acquiring sounds around the electronic device 100. The microphone 142 may receive input of an external acoustic signal and generate electronic voice information. The microphone 142 may use various noise elimination algorithms to eliminate the noise generated in the process of receiving the input of an external acoustic signal. The image information or voice information input by the camera 141 or the microphone 142 may be input as an input value of an artificial intelligence model.

[0107] The touch panel 143 may include various circuits and receive inputs of various user inputs. The touch panel 143 may receive inputs of data through user operations. In addition, the touch panel 143 may be included in combination with a display to be described later.

[0108] The inputter 140 may be various components that receive input of various data in addition to the aforementioned camera 141 , microphone 142 , and touch panel 143 .

[0109] The display 150 may include components for outputting various images. The display 150 for providing various images may be implemented as a display panel in various forms. For example, the display panel may be implemented as various display technologies, such as, but not limited to, a liquid crystal display (LCD), an organic light emitting diode (OLED), an active matrix organic light emitting diode (AM-OLED), a liquid crystal on silicon (LCOS), a digital light processing (DLP), etc. The display 150 may be coupled to at least one of the front surface area, the side surface area, or the rear surface area of ​​the electronic device 100 in the form of a flexible display.

[0110] The audio output 160 may include various audio output circuits, which are not only used to output various audio data that have been subjected to various processing operations (such as decoding or amplification, and noise filtering performed by an audio processor), but also used to output various notification sounds or voice messages. The audio processor is a component that performs audio data processing. At the audio processor, various processes such as decoding or amplification and noise filtering of audio data may be performed. The audio data processed at the audio processor 150 may be output to the audio output 160. In particular, the audio output may be implemented as a speaker, but this is merely an example, and the audio output may be implemented as an output terminal capable of outputting audio data.

[0111] As described above, the processor 120 may include various processing circuits and control the overall operation of the electronic device 100. The processor 120 may include a RAM 121, a ROM 122, a main CPU 124, a graphic processor 123, first to nth interfaces 125-1 to 125-n, and a bus 126. The RAM 121, the ROM 122, the main CPU 124, the graphic processor 123, the first to nth interfaces 125-1 to 125-n, etc. may be connected to each other through the bus 126.

[0112] The ROM 122 stores a set of instructions for system booting, etc. When a power-on instruction is input and power is supplied, the main CPU 124 copies an O / S stored in a memory in the RAM 121 according to the instructions stored in the ROM 122, and boots the system by executing the O / S. When the booting is completed, the main CPU 124 copies various types of applications stored in a memory in the RAM 121, and performs various types of operations by executing the applications copied in the RAM 121.

[0113] The main CPU 124 accesses the memory 110 and performs booting using the O / S stored in the memory 110. The main CPU 124 uses various types of programs, contents, data, etc. stored in the memory 110 to perform various operations.

[0114] The first to nth interfaces 125-1 to 125-n may be connected with the above-mentioned various types of components. One of the interfaces may be a network interface connected with an external device through a network.

[0115] Figure 5a is a flow chart illustrating an exemplary method for providing a response to a user utterance according to an embodiment of the present disclosure.

[0116] When a user utterance is input, in operation S510, the electronic device may determine a task associated with the input user utterance. For example, the electronic device 100 may obtain the entity and intention of the user utterance input through the natural language understanding part 420, and determine the task for the user utterance. According to an embodiment of the present disclosure, the electronic device 100 may obtain the dialogue action, the main action and the entity in the user utterance through the natural language understanding part 420. The dialogue action may refer to, for example, an intentional action of a speaker for the purpose of executing a dialogue included in the utterance, and indicates whether the user utterance is a request for an action (Request), a request for a certain variable value of the listener by the speaker (WH-Question), or a request for a yes / no (Y / N) answer of the listener by the speaker (YN-Question), or providing information (notification) to the listener through the speaker, etc. In addition, the main action refers to semantic information, which indicates the desired action through the utterance of the dialogue in a specific domain. In addition, the entity is information added to specify the meaning of the action expected in a specific domain.

[0117] For example, in the case where the user utterance is "What's the weather like today?", in the case where the dialog action of the user utterance may be "WH-Question", the primary action may be "Weather check", and the entity may be "Today". Therefore, the task for the user utterance may be determined to be checking today's weather and providing a response.

[0118] The electronic device 100 may determine additional information that is not included in the user utterance but is associated with the main action as an entity. Information about which additional information is associated with the main action may be pre-stored in the electronic device 100. For example, in a case where the additional information associated with "weather check" as the main action is the current location of the electronic device 100, the electronic device 100 may determine the current location of the electronic device 100 (e.g., "Yangjae-dong") as an entity even though it is not included in the user utterance "What's the weather like today?" The electronic device 100 may determine that the task for the user utterance is to check the weather in Yangjae-dong today, and provide a response based on the entity "today" obtained from the user utterance and the entity Yangjae-dong obtained from the additional information associated with the main action.

[0119] In operation S520, the electronic device 100 may determine additional tasks related to the determined task. Specifically, the electronic device 100 may determine additional tasks related to the determined task based on, for example, past user utterance history information and user preference information. For example, in the case where the determined task is a weather check task, the electronic device 100 may determine a temperature check task and an air pollution check task as additional tasks based on the user utterance history and the user preference information.

[0120] In operation S530, the electronic device 100 may determine additional response information based on information about the degree of association between the determined task and the additional task. For example, the electronic device 100 may determine a task with a degree of association greater than a threshold value as an additional task among multiple additional tasks related to the determined task. For example, if the task determined according to the user's utterance is a temperature check, and the degree of association between the temperature check task and the weather check task is 0.9, and the degree of association between the temperature check task and the atmospheric pollution check task is 0.5, and the threshold is 0.7, the electronic device 100 may determine the weather check task with a degree of association greater than 0.7 with the temperature check task as an additional task, and may not determine the atmospheric pollution check task with a degree of association less than 0.7 with the temperature check task as an additional task. The threshold value may be determined by various methods. As an example, the threshold value may be determined by user settings. As another example, the threshold value may be output by inputting the user's utterance and the response to the user's utterance as input values ​​into a trained artificial intelligence model.

[0121] In addition, in operation S540, the electronic device 100 may provide response information and additional response information for the determined task. For example, the natural language generation part 440 of the electronic device 100 may generate a response to the user's utterance as a natural language based on the response information and the additional response information for the determined task, and provide the response. However, the present disclosure is not limited thereto, and the electronic device 100 may provide the response information and additional response information for the determined task in the form of text.

[0122] If a response to the user utterance is provided, in operation S550, the electronic device 100 may update the knowledge database 460 based on the input user utterance and the additional response information for the user utterance. For example, the electronic device 100 may input the input user utterance and the additional response information for the user utterance as input values ​​into the trained artificial intelligence model, and update the information about the association between the plurality of tasks, and store the updated information about the association in the knowledge database 460. As another example, the electronic device 100 may input the input user utterance and the additional response information for the user utterance as input values ​​into the trained artificial intelligence model, and update the threshold value for the task related to the user utterance, and update the updated threshold value in the knowledge database 460.

[0123] Figure 5b is a diagram illustrating an exemplary method for providing an additional response according to another embodiment of the present disclosure.

[0124] For example, the electronic device 100 may provide response information to the user's utterance and provide a query message for providing additional response information. For example, the electronic device 100 may not provide response information and additional response information to the user's utterance at the same time, but may provide a query message to inquire whether to provide additional response information. For example, if the user utters "What is the temperature in Seoul today?" as input, the electronic device 100 may provide a response to the user's utterance and a query message, such as "The temperature in Seoul today is 20 degrees. Do you still want to know the weather in Seoul?" If a user command for a query message is received, the electronic device 100 may provide an additional response to the user command. That is, the electronic device 100 may provide a response to the user's utterance and a query message for providing additional response information, such as "Do you still want to know the weather in Seoul?" If the electronic device 100 receives a user command of "Yes" to the user's query "Do you still want to know the weather in Seoul?", the electronic device 100 may provide an additional response, such as "The weather in Seoul is good today."

[0125] Figure 6a is a diagram showing an exemplary method for updating information related to the degree of association between tasks according to an embodiment of the present disclosure, and Figure 6b is a diagram illustrating an example of information related to the degree of association between tasks stored in the knowledge database 460 according to an embodiment of the present disclosure.

[0126] like Figure 6a As shown, the electronic device 100 may update information on the degree of association between the temperature check task, the weather check task, the air pollution check task, and the population density check task based on the user utterance history and the user preference information.

[0127] For example, Figure 6aAs shown on the left side of , in the initial state, the association between each task may be 0. In this example, the electronic device 100 may provide temperature information for a user utterance requesting a temperature check, weather information for a user utterance requesting a weather check, air pollution information for a user utterance requesting an air pollution check, and population density information for a user utterance requesting a population density check.

[0128] When user utterances and response information to the user utterances are accumulated in the knowledge database 460, the electronic device 100 may update the information of the degree of association between each task based on the accumulated data. For example, the electronic device 100 may update the information of the degree of association between each task based on the information of the associated tasks for each same entity in the user utterance. For example, if a user utterance related to multiple tasks (temperature check, weather check, air pollution check, etc.) for the same entity (place) is input, the electronic device 100 may update the information related to the degree of association between each of the multiple tasks for the same entity.

[0129] As another example, if there is an additional user utterance for a response to a user utterance, the electronic device 100 may update the information of the degree of association between each task based on the information of the task for the user utterance and the information of the task for the additional user utterance. For example, if there is a response to a user utterance and an additional user utterance, the electronic device 100 may increase the degree of association between the task for the user utterance and the task for the additional user utterance. For example, after the electronic device 100 provides a response notifying the temperature of the user utterance requesting a temperature check, if an additional user utterance requesting a weather check is input, the electronic device 100 may increase the degree of association between the temperature check task and the weather check task. The electronic device 100 may update the task information in consideration of the input order of the user utterance and the additional user utterance. For example, if the task for the user utterance and the task for the additional utterance are input in the order of the temperature check task, the weather check task, and the atmospheric pollution check task, the electronic device 100 may set the degree of association between the temperature check task and the weather check task to be higher than the degree of association between the temperature check task and the atmospheric pollution check task.

[0130] As another example, the electronic device 100 may update the information in each task based on feedback for the response to the user utterance. For example, while providing a response to the user utterance, if a user command to stop responding is input, the electronic device 100 may reduce the association between the task corresponding to the response to be provided after the user command to stop responding is input and the task corresponding to the user utterance. As an example, it can be assumed that the electronic device 100 will provide a response to the user utterance "What's the temperature today?" is "Today's temperature is 20 degrees and the weather is good." In the case where the electronic device 100 provides a response about today's temperature, and while the electronic device 100 provides a response about the weather, a user command to stop responding and cannot provide a response about the weather is input, the electronic device 100 may reduce the association between the temperature check task and the weather check task.

[0131] The electronic device 100 may update the degree of association between tasks based on the user utterance history and the user preference information, and may also update a threshold for determining an additional task.

[0132] In summary, the electronic device 100 may update information about the degree of association between each task and store the information in the knowledge database 460 based on the historical information of the user's utterance and the user's preference information, such as Figure 6a For example, Figure 6b As shown, according to the various embodiments of the present disclosure as described above, information about the correlation between "weather" and "good / cloudy..., UV rays, temperature, humidity, air quality and tomorrow's weather", information about the correlation between "tomorrow's weather" and "tomorrow's temperature, tomorrow's air quality, and tomorrow's humidity", and information about the correlation between "place" and "travel place, weather, accommodation, transportation, famous restaurants and city information" can be stored in the knowledge database 460.

[0133] Figure 7 is a diagram illustrating an example of providing a response to a user utterance according to an embodiment of the present disclosure.

[0134] like Figure 7 As shown, if a user utterance is input, the electronic device 100 may analyze the user utterance and generate a response to the user utterance. The electronic device 100 may store the user utterance and information related to the user utterance in the form of a table. For example, the stored table may include information on query topics (e.g., weather information, traffic schedules, etc.), query frequencies (e.g., n times per day, etc.), average time of continuous queries (time of additional queries regarding user utterances), and detailed contents of the queries.

[0135] For example, the electronic device 100 may not only store the response to the user utterance, but also match additional user utterances and response information about the additional user utterances and store them. If a user utterance is input after obtaining a table for various user utterance histories, the electronic device 100 may determine the response to the user utterance based on the table for various user utterance histories.

[0136] Figure 8a , Figure 8b , Fig. 9 , Fig.10 ,and Fig.11 is a diagram illustrating an example in which a response to a user utterance is of a query type according to an embodiment of the present disclosure.

[0137] Figure 5a , Figure 5b , Figure 6, and Figure 7 An example is described in which the response type of the electronic device 100 to the user utterance is a response providing type, but the present disclosure is not limited thereto. For example, in the case where there is an unclear part in the user utterance, the electronic device 100 may not provide an answer to the user utterance, but may provide a response to a query about the insufficient part of the user utterance. For example, if as a result of analyzing the user utterance, there is an omitted part in an entity or a task, or there are multiple tasks for the user utterance, the electronic device 100 may provide a query requesting the omitted task or entity, or provide a query asking which task to select from multiple tasks. For example, when a user utterance such as "What's the weather like today?" is input, the electronic device 100 may determine to omit an entity related to a location, and provide a response such as "Do you want to know today's weather?" In the case of a user utterance such as "Call Saebom" being input (see, for example, Figure 8a ), the electronic device 100 may determine that there are multiple searches for “Saebom” and provide a response such as “Who should I call between Saebom Choi, Saebom Lee, and Saebom Park?”

[0138] If the response type to the user utterance is a query providing type, the electronic device 100 may provide a selection type for receiving a selection of one of multiple alternatives for the user utterance, or provide one of a Yes / No type of query as to which of the multiple alternatives is correct, as a response to the user utterance.

[0139] For example, if a user utterance such as "Call Saebom" is input, the electronic device 100 may determine that the response to the user utterance is a query providing type. Figure 8aAs shown, the electronic device 100 may provide a response to the user's utterance as a selection type for receiving a selection of one of a plurality of alternatives (Saebom Choi, Saebom Lee, and Saebom Park), such as "Three Saeboms were searched. Who should I call among Saebom Choi, Saebom Lee, and Saebom Park?". Figure 8b As shown, the electronic device 100 may provide a response to the user's utterance as a "yes / no" type of determining one of a plurality of alternatives, and inquiring whether the determined alternative is correct, such as "Should I call Saebom Lee?".

[0140] The electronic device 100 may decide whether to provide a response as a selection type, or to provide a response as a yes / no type through various methods. As an example, the electronic device 100 may decide the response provision type according to the user setting. For example, if the user setting provides a response as a selection type, the electronic device 100 may provide a response as a selection type, and if the user setting provides a response as a yes / no type, the electronic device 100 may provide a response as a yes / no type. As another example, the electronic device 100 may determine a task for the user utterance, determine a plurality of alternatives for the determined task, and decide a type between the selection type and the yes / no type based on information about the association between the determined task and the plurality of alternatives and a threshold. For example, if a user utterance such as "Call Saebom" is input, the electronic device 100 may obtain a call sending task for the user utterance, and obtain a plurality of alternatives (Saebom Choi, Saebom Lee, and Saebom Park) for the object to send a call. In the case where the association between the call sending task and Saebom Choi is 0.1, the association between the call sending task and Saebom Lee is 0.9, the association between the call sending task and Saebom Park is 0.5, and the threshold is 0.7, because there is an alternative (Saebom Lee) greater than the threshold, the electronic device 100 can provide a response such as "Should I call Saebom Lee?" (yes / no type). In the case where the association between the call sending task and Saebom Choi is 0.1, the association between the call sending task and Saebom Lee is 0.9, and the association between the call sending task and Saebom Park is 0.5, and the threshold is 0.3, because there are multiple alternatives (Saebom Lee, Saebom Park) greater than the threshold, the electronic device 100 can provide a response such as "Who should I call between Saebom Lee and Saebom Park?" (selection type). As another example, when the association between the call sending task and Saebom Choi is 0.1, and the association between the call sending task and Saebom Lee is 0.9, and the association between the call sending task and Saebom Park is 0.5, and the threshold is 0.3, the electronic device 100 may provide a response such as "Who should I call between Saebom Choi, Saebom Lee, and Saebom Park?" (selection type). For example, if there are multiple alternatives greater than the threshold, the electronic device 100 may provide an alternative (Saebom Park) less than the threshold as a response.

[0141] Fig. 9 is a flow chart illustrating an exemplary method for providing a query-type response to a user utterance according to an embodiment of the present disclosure.

[0142] When a user utterance is input, in operation S910, the electronic device 100 may determine a task related to the input user utterance. For example, the electronic device 100 may obtain an entity and a task of the user utterance input through the natural language understanding part 420, and determine a task for the user utterance. As an example, in the case where the user utterance is "Call Saebom", the determined task may be to send a call.

[0143] In operation S920, the electronic device 100 may determine an additional query to be included in the determined response information based on the determined task. For example, there may be multiple pieces of information necessary for performing the task. For example, in the case where there are multiple pieces of information necessary for performing the task, the electronic device 100 may determine an additional query for determining one piece of information among the multiple pieces of information. For example, the additional query may be a selection type, such as "Three Saeboms were searched. Who should I call between Saebom Choi, Saebom Lee, and Saebom Park?", or a yes / no type, such as "Should I call Saebom Lee?"

[0144] In operation S930, the electronic device 100 may determine additional response information based on information about the association between the determined task and the additional query. For example, the electronic device 100 may determine a task for the user's utterance, determine multiple pieces of information for the determined task, and determine a selection type or a type between the yes / no types based on information about the association between the determined task and the additional query determined based on multiple alternatives and a threshold. As described above, in the case where there is one additional query greater than the threshold, the electronic device 100 may decide that the yes / no type is additional information, and in the case where there are multiple additional queries greater than the threshold, the electronic device 100 may decide that the selection type is additional queries.

[0145] In operation S940, the electronic device 100 may provide response information and additional queries (or additional responses) for the determined task. As an example, the natural language generation portion 440 of the electronic device 100 may generate a response to the user's utterance as a natural language based on the response information and the additional response information for the determined task, and provide the response. However, the present disclosure is not limited thereto, and the electronic device 100 may provide the response information and the additional response information for the determined task in the form of text.

[0146] In operation S950, in the case of providing a response to the user utterance, the electronic device 100 may update the knowledge database 460 based on the input user utterance and the additional response information (e.g., additional query) for the user utterance. For example, the electronic device 100 may input the input user utterance and the additional response information for the user utterance as input values ​​into the trained artificial intelligence model, and update the information about the association between the plurality of tasks, and store the updated information about the association in the knowledge database 460. As another example, the electronic device 100 may input the input user utterance and the additional response information for the user utterance as input values ​​into the trained artificial intelligence model, and update the threshold of the task related to the user utterance, and update the threshold updated in the knowledge database 460.

[0147] Fig.10 is a diagram showing an example of providing a response to a user utterance according to yet another embodiment of the present disclosure.

[0148] like Fig.10 As shown, when a user utterance is input, the electronic device 100 may analyze the user utterance and generate a query about the user utterance. When an additional user utterance for a query about the user utterance is input, the electronic device 100 may provide a response to the additional user utterance. The electronic device 100 may store the user utterance and information related to the user utterance in the form of a table. For example, the stored table may include information about the query subject (corresponding part of the call, method of sending a message, search engine, etc.), query type (selection type, yes / no type, etc.), response type (selection type, yes / no type, etc.), response content, information about the query generation time, etc.

[0149] For example, the electronic device 100 may match additional queries about user utterances, response types for the additional queries, and response contents to the additional queries, and store them. When a user utterance is input after obtaining a table of various user utterance histories, the electronic device 100 may decide a response to the user utterance based on the table of various user utterance histories.

[0150] In the case where the above-mentioned various operations of the electronic device 100 are performed by the artificial intelligence model, the electronic device 100 may operate the artificial intelligence model, but the present disclosure is not limited thereto. Fig.11 As shown, operations of the artificial intelligence model may be performed at server 200 .

[0151] For example, when a user utterance is input, the electronic device 100 may send the input user utterance to the server 200. The server 200 may determine a task related to the user utterance based on the received user utterance, determine a response mode including a task related to the user utterance and an additional task related to the determined task based on the determined task, obtain response information for the task related to the user utterance and additional response information for the additional task, generate a natural language based on the obtained response information and the additional response information, and send the natural language to the electronic device 100.

[0152] According to an embodiment, the electronic device 100 and the server 200 may be connected to each other through remote communication.

[0153] The electronic device 100 and the server 200 may be connected to each other via short-range communication (e.g., Wi-Fi, Wi-Fi Direct, Bluetooth). The server 200 may be, for example, a device located in the user's home. For example, the server 200 may additionally operate an artificial intelligence model for the electronic device 100 while performing separate functions in the user's home, such as, but not limited to, a TV, a refrigerator, an air conditioner, an AI speaker, etc.

[0154] According to an embodiment, the server 200 may store several components of the artificial intelligence model. For example, the server 200 may store Figure 4 The server 200 may store a separate knowledge database 460 for each user, and based on the user information of the electronic device 100 connected to the server, send the user's corresponding knowledge database 460 to the electronic device 100. The server 200 may send the knowledge database 460 to the electronic device 100 at regular intervals, or send it to the electronic device 100 whenever the knowledge database 460 of the server 200 is updated.

[0155] On the other hand, Figure 8a , Figure 8b , Fig. 9 , Fig.10 ,and Fig.11, an electronic device 100 is described that provides an additional query for obtaining additional information when there is an unclear part in the user's utterance. However, even if there is an unclear part in the user's utterance, the electronic device 100 can obtain additional information based on user context information and provide a response to the user. For example, in the case where the user utterance is "How's the weather?", the electronic device 100 can determine that the location for requesting a weather check is unclear. The electronic device 100 can use GPS information, etc. to determine the current location of the electronic device 100 and provide weather information for the determined location, rather than providing the user with an additional query for requesting a weather check location. As another example, where the user utterance is "Call Saebom", instead of providing another query, such as "Who should I call between Saebom Choi, Saebom Lee, and Saebom Park?" or "Should I call Saebom Choi?", the electronic device 100 can determine that there were no logs called Saebom Lee and Saebom Park in the past, and call Saebom Choi.

[0156] Fig.12 is a flowchart illustrating an exemplary method of controlling an electronic device according to an embodiment of the present disclosure.

[0157] In operation S1210, the electronic device 100 may receive a user utterance. In operation S1220, the electronic device 100 may determine a task related to the user utterance based on the received user utterance. For example, in the case where the user utterance is "What's the weather like today?", the electronic device 100 may determine "weather check" as a task related to the user utterance. As another example, in the case where the user utterance is "Call Saebom", the electronic device 100 may determine "send call" as a task related to the user utterance.

[0158] In operation S1230, the electronic device 100 may determine the task related to the user utterance and the additional task related to the task based on the task related to the user utterance. For example, the electronic device 100 may determine the task whose relevance to the task related to the user utterance is greater than a threshold as the additional task. As described above, the relevance and threshold of the task related to the user utterance may be determined based on the user utterance history and the user preference information.

[0159] In operation S1240, the electronic device 100 may acquire response information for a task related to the user utterance and response information for an additional task. In operation S1250, the electronic device 100 may provide the acquired response information and the additional response information.

[0160] Fig.13is a flowchart illustrating an exemplary method of controlling an electronic device according to another embodiment of the present disclosure.

[0161] Reference Fig.13 In operation S1310, the electronic device 100 may receive an input of a first utterance of a user. The first utterance may be, for example, "What is the temperature in Umyeon-dong?".

[0162] In operation S1320, the electronic device 100 may identify a first task for the first utterance based on the first utterance. The first task may be, for example, a weather check for Umyeon-dong.

[0163] In operation S1330, the electronic device 100 may provide a response for the first task according to a predetermined response mode. For example, the response for the first task provided according to the predetermined response mode may be "The weather in Umyeong-dong is good and warm today."

[0164] In operation S1340, the electronic device 100 may receive an input of a second utterance of the user. For example, the second utterance may be "Tell me about the concentration of fine dust in Umeyon-dong".

[0165] In operation S1350, the electronic device 100 may identify a second task for the second utterance based on the second utterance, and determine a degree of association between the first task and the second task.

[0166] Separately from the determination of the association degree, the electronic device 100 may provide a response according to a predetermined response pattern regarding the second task. For example, the response may be "the concentration of fine dust in Umyeon-dong is 10, and the concentration of ultrafine dust is 20".

[0167] If the degree of association determined in operation S1350 satisfies a predetermined condition, in operation S1360, the electronic device 100 may set a response mode for the first task based on the second task.

[0168] According to an embodiment, a response mode for the first task may be set so that the response mode for the second task can additionally reflect the response mode for the first task. After the response mode for the first task is set based on the second task as described above, if the first utterance is input, the electronic device may provide a response according to the set response mode. For example, when the first utterance, i.e., "What is the temperature in Umyeon-dong?", is input again, the response mode for the second task is additionally reflected, and a response may be provided according to the response mode set for the first task, such as "Today's weather in Umeyon-dong is good and warm, the fine dust concentration is 10, and the ultrafine dust concentration is 20".

[0169] According to another embodiment, a response mode for a second task may be set so that the response mode for the first task can be additionally reflected to the response mode for the second task. After the response mode for the second task is set based on the first task as described above, if a second utterance is input, the electronic device may provide a response according to the set response mode. For example, when the second utterance, i.e., "What is the fine dust level in Umyeon-dong?", is input again, the response mode for the first task is additionally reflected, and a response may be provided according to the response mode set for the second task, such as "Today's weather in Umeyon-dong is good and warm, the fine dust concentration is 10, and the ultrafine dust concentration is 20".

[0170] According to the above exemplary embodiment, a response is not provided according to a fixed response mode, but the response mode can be adaptively changed according to the user's habits, preferences, etc. reflected as the user uses the electronic device. Therefore, there is an effect that a response that better suits the user's intention can be provided, and the user can obtain a desired response at one time without querying several times, etc.

[0171] Functions related to artificial intelligence according to the present disclosure may be operated through the processor 120 and the memory 110 .

[0172] The processor 120 may include, for example, one or more processors 120. The one or more processors 120 may include, for example, but not limited to, a general-purpose processor such as a CPU, an AP, etc., a graphics-specific processor such as a GPU, a VPU, etc., an artificial intelligence processor such as an NPU, etc.

[0173] The one or more processors 120 may control the electronic device 100 to process input data according to a predetermined operation rule or artificial intelligence model stored in the memory 110. The predefined operation rule or artificial intelligence model is characterized in that it is made through training.

[0174] "Made by learning" may refer to, for example, a predetermined operating rule or artificial intelligence model of a desired feature generated by applying a learning algorithm to a plurality of learning data. Such learning may be performed in the device itself that performs artificial intelligence according to the present disclosure, or may be performed by a separate server / system.

[0175] The artificial intelligence model may include multiple neural network layers. Each layer may have multiple weight values, and the calculation of the layer may be performed by the calculation result of the previous layer and the calculation of multiple weight values. Examples of neural networks include, for example, but not limited to, convolutional neural networks (CNNs), deep neural networks (DNNs), recurrent neural networks (RNNs), restricted Boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), deep Q networks, etc., and the neural networks in the present disclosure are not limited to the above examples.

[0176] A learning algorithm may refer to, for example, a method of using a plurality of learning data to train a predetermined target device (e.g., a robot) so that the predetermined target device makes a decision or makes a prediction by itself. Examples of learning algorithms include, but are not limited to, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, and the learning algorithms in the present disclosure are not limited to the above examples unless specified.

[0177] The term "part" or "module" used in the present disclosure includes a unit including hardware, software, firmware or any combination thereof, and it can be used interchangeably with terms such as logic, logic block, component or circuit. In addition, a "component" or "module" can be a component including an integrated body or a minimum element or a part thereof that performs one or more functions. For example, a module may include an application specific integrated circuit (ASIC).

[0178] Various embodiments of the present disclosure may be implemented as software including instructions stored in a machine-readable storage medium, which can be read by a machine (e.g., a computer). A machine may refer to, for example, a device that calls instructions stored in a storage medium and can operate according to the called instructions, and the device may include an electronic device (e.g., electronic device 100) according to the aforementioned embodiment. In the case where the instructions are executed by a processor, the processor may perform the function corresponding to the instructions by itself, or use other components under its control to perform. The instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. A "non-transitory" storage medium may not include a signal and is tangible, but does not indicate whether the data is stored in the storage medium semi-permanently or temporarily.

[0179] According to an embodiment of the present disclosure, the method according to the various embodiments described in the present disclosure may be provided in the case of being included in a computer program product. A computer program product refers to a product that can be traded between a seller and a buyer. The computer program product can be distributed online in the form of a machine-readable storage medium (e.g., a compact disc read-only memory (CD-ROM)), or distributed through an application store (e.g., Play StoreTM). In the case of online distribution, at least a portion of the computer program product may be stored in a storage medium, such as a manufacturer's server, an application store's server, and a memory of a relay server, at least temporarily, or may be temporarily generated.

[0180] In addition, each of the components (e.g., modules or programs) according to the aforementioned various embodiments may include a single object or multiple objects. In the above-mentioned corresponding subcomponents, some subcomponents may be omitted, or other subcomponents may be further included in various embodiments. Generally or additionally, some components (e.g., modules or programs) may be integrated into objects, and the functions performed by each component in the same or similar manner before integration are performed. The operations performed by the modules, programs or other components according to various embodiments may be performed sequentially, in parallel, repeatedly or tentatively. Alternatively, at least some operations may be performed or omitted in different orders, or other operations may be added.

[0181] Although various exemplary embodiments have been shown and described, it should be understood that the various exemplary embodiments are intended to be illustrative rather than restrictive. Therefore, those skilled in the art will appreciate that various changes in form and details may be made without departing from the true spirit and full scope of the present disclosure, including, for example, the attached claims and their equivalents.

Claims

1. A method for controlling an electronic device, comprising: Receiving a first utterance input; identifying a first task for the first utterance based on the first utterance; providing a response to the first task based on a predetermined response mode set for the first task; receiving input of a second vocalization after providing a response to the first task; identifying a second task for the second utterance based on the second utterance; providing a response to the second task based at least on a predetermined response mode set for the second task; determining a correlation between the first task and the second task; as well as At least based on the determined association between the first task and the second task satisfying a predetermined condition, a response mode for the first task is set based on the second task.

2. The method according to claim 1, wherein: The response pattern is determined based on at least one of information related to a length of a response to an utterance or information related to a type of information included in the response to the utterance.

3. The method according to claim 1, wherein: The predetermined response mode set for the first task includes a response mode selected by a command or a response mode set automatically.

4. The method according to claim 3, further comprising: Inputting speech based on the first utterance into a trained artificial intelligence model to obtain information related to acoustic features of the first utterance; as well as identifying a user based on the information obtained related to the acoustic features, Wherein, a predetermined response mode set for the first task is determined based on the identified session history and the preference information of the user.

5. The method according to claim 1, wherein: The predetermined condition includes a condition that information related to the degree of association between the first task and the second task is equal to or greater than a threshold value.

6. The method according to claim 1, further comprising: determining, based on receiving an input of a third utterance, whether a third task for the third utterance is associated with the first task; as well as A response is provided based on the set response mode for the first task.

7. The method according to claim 1, further comprising: receiving an input of a third utterance; as well as identifying a third task for the third utterance based on the third utterance, and determining a degree of association between the first task and the third task, and The settings also include: determining priorities of the second task and the third task based on the determined association between the first task and the third task satisfying a predetermined condition; and based on the determined priorities, the second task and the third task, A response mode for the first task is set.

8. The method according to claim 5, further comprising: Get feedback on the responses provided; as well as The degree of association between the first task and the second task is updated based on the acquired feedback.

9. The method according to claim 1, further comprising: Based on the degree of association between the first task and the second task, the first task and the second task are stored in the form of an ontology.

10. The method according to claim 1, in, performing at least one of identifying the first task, determining the relevance, setting a response mode, or providing a response through an artificial intelligence model, and The artificial intelligence model includes a plurality of layers each including at least one node, and each of the at least one node includes a neural network model having connection weights for interpreting an input value.

11. Electronic devices, including: a memory configured to store at least one command; as well as a processor configured to execute the at least one command, Wherein, the processor is configured to control the electronic device to: receiving an input of a first utterance, identifying a first task for the first utterance based on the first utterance, providing a response to the first task based on a predetermined response mode set for the first task, receiving a second utterance input after providing a response to the first task, identifying a second task for the second utterance based on the second utterance, and providing a response to the second task based on a predetermined response mode set for the second task; A degree of association between the first task and the second task is determined, and a response mode for the first task is set based on the second task at least based on the determined degree of association between the first task and the second task satisfying a predetermined condition.

12. The electronic device according to claim 11, wherein: The response pattern is determined in the context of including at least one of information related to a length of a response to an utterance or information related to a type of information included in the response to the utterance.

13. The electronic device according to claim 11, wherein: The predetermined response mode includes a response mode selected by a command or a response mode automatically set.

14. The electronic device according to claim 11, in, The processor is further configured to control the electronic device to: input the speech based on the first utterance into a trained artificial intelligence model to obtain information related to acoustic features of the first utterance, and identify the user based on the obtained information related to the acoustic features, and Wherein, the predetermined response mode is determined based on the identified session history and the preference information of the user.

15. The electronic device according to claim 11, wherein: The predetermined condition includes a condition that information related to the degree of association between the first task and the second task is equal to or greater than a threshold value.

Images