Electronic device and control method thereof

Abstract

An electronic device and a control method thereof are disclosed. The control method for an electronic device according to the disclosure includes: when a user command is input during execution of an application, acquiring information about the application being executed and storing the same in a memory; and inputting a user question related to the application into a learned artificial intelligence model so as to output a reply to the acquired user question when a user question related to the application is input, wherein the reply to the user question can be determined based on the information of the executed application.

Description

Technical Field

[0001] This disclosure relates to an electronic device and a control method thereof. More specifically, this disclosure relates to an electronic device and a control method thereof capable of providing optimized responses to user questions based on a personalized database. Background Technology

[0002] Artificial intelligence (AI) systems are computer systems that enable machines to become intelligent through self-learning and recognition. Because AI systems have improved recognition rates, they are being used more widely and can better understand user preferences, leading to the gradual replacement of existing rule-based intelligent systems by deep learning-based AI systems.

[0003] Artificial intelligence technology consists of machine learning (deep learning) and the elemental technologies that utilize machine learning.

[0004] Machine learning can be an algorithmic technique that classifies / learns features of input data, and element techniques can be techniques that utilize machine learning algorithms such as deep learning, and can be included in technical fields such as language understanding, visual understanding, inference / prediction, knowledge representation, and motion control.

[0005] The following describes various fields where artificial intelligence technologies can be applied. Language understanding is the technology of recognizing and applying / processing human language / characters, and includes natural language processing, machine translation, dialogue systems, question answering, speech recognition / synthesis, etc. Visual understanding is a technology for processing things that humans visually recognize, and includes object recognition, object tracking, image search, human recognition, scene understanding, spatial understanding, image enhancement, etc. Inference and prediction are technologies for determining information through logical inference and prediction, and include knowledge-based / likelihood-based inference, optimization prediction, preference-based planning, recommendation, etc. Knowledge representation is a technology for automatically processing human experience information into knowledge data, and includes knowledge construction (generating / classifying data), knowledge management (utilizing data), etc. Motion control is a technology used to control the autonomous driving of vehicles and the movement of robots, and includes motion control (navigation, collision, driving), manipulation control (behavior control), etc.

[0006] In many cases, artificial intelligence (AI) technologies can operate on external servers. However, since the data used in AI technologies is often personal information, security-related issues may arise. Summary of the Invention

[0007] Technical issues

[0008] This disclosure provides an electronic device and its control method capable of providing optimized responses to user questions based on a personalized database.

[0009] Technical solution

[0010] According to an embodiment, the control method for an electronic device includes: obtaining at least one of the following based on a user command input while running an application: the type of the application, content information related to the running application, and functions used by the application, and storing them in a memory; inputting a user question related to the application into a trained artificial intelligence model based on an input user question related to the application, and outputting a response to the input user question; and determining a response to the user question based on the type of the application, content information related to the running application, and functions used by the application.

[0011] Storage may include obtaining the application type, content information related to the running application, and at least one of the functions used by the application and storing them in a database.

[0012] Storage can include text based on running applications, extracting key information from the text and storing it.

[0013] The storage may include storing historical information based on detected user interactions that perform functions on a running application. The historical information includes information about the functions performed and information about user interactions.

[0014] The output may include at least one of providing search results for the user's question, performing an action on the user's question, or performing an application function on the user's question.

[0015] The output can include questions related to the content search based on user questions, and recommendations for applications that can be used to replay the searched content based on application-related information.

[0016] The output may include sending the user question to an external server based on the input user question, receiving response candidates from the external server, and outputting a response to the user question based on the received response candidates and information stored in the database.

[0017] The storage may include storing at least one of the following: domain information and application type of a running application, content information related to the running application, and functions used by the application; and the artificial intelligence model is characterized by obtaining answers to user questions based on the stored domain information and application type, content information related to the running application, and functions used by the application.

[0018] User questions can be either spoken or written by the user.

[0019] According to one embodiment, the electronic device may include a memory and a processor configured to run an application stored in the memory. The processor obtains and stores in the memory at least one of the following based on user commands input when running the application: the type of the application, content information related to the running application, and functions used by the application. Based on user questions related to the application, the processor inputs user questions related to the application into a trained artificial intelligence model and outputs a response to the input user questions.

[0020] The processor can be configured to acquire at least one of the following: the type of the application, content information related to the running application, and functions used by the application, and store them in memory.

[0021] The processor can be configured to extract key information from running applications, including text, and store it in memory.

[0022] The processor can be configured to store historical information in memory based on detected user interactions that perform functions on a running application. The historical information includes information about the functions performed and information about the user interactions.

[0023] The processor can be configured to output at least one of the following: providing search results for the user's question, performing an action on the user's question, and performing application functions on the user's question.

[0024] The processor can be configured to recommend applications that replay the searched content based on user-related questions or questions relevant to the content search.

[0025] The electronic device may also include a communicator, and the processor may be configured to send a user question to an external server via the communicator based on an input user question, receive candidate responses to the user question from the external server via the communicator, and output a response to the user question based on the received candidate responses and information stored in memory.

[0026] The processor can be configured to store in memory at least one of the following: domain information about the running application and the type of the application, content information related to the running application, and functions used by the application; and the artificial intelligence model can be configured to obtain an answer to a user's question based on the stored domain information and the type of the application, the content information related to the running application, and at least one of the functions used by the application.

[0027] User questions can be either spoken or written by the user.

[0028] Invention Effects

[0029] According to various implementation methods, electronic devices can use artificial intelligence models to simultaneously protect user privacy and output customized responses for the user. Attached Figure Description

[0030] Figure 1 This is an exemplary view that briefly illustrates the operation of an electronic device according to this disclosure;

[0031] Figure 2 This is a block diagram that briefly illustrates the configuration of an electronic device according to an embodiment of the present disclosure;

[0032] Figure 3 This is a block diagram showing in detail an electronic device according to embodiments of the present disclosure;

[0033] Figure 4 This is a block diagram illustrating a dialogue system according to an embodiment of the present disclosure;

[0034] Figures 5A and 5B are exemplary views illustrating data stored in a knowledge database according to embodiments of the present disclosure;

[0035] Figure 6 This is an exemplary view illustrating a method for classifying domains (or categories) of data stored in a knowledge database according to embodiments of the present disclosure;

[0036] Figure 7 This is an exemplary view illustrating a domain management method according to an embodiment of the present disclosure;

[0037] Figure 8 This is an exemplary view illustrating a method for obtaining data stored in a knowledge database according to embodiments of the present disclosure;

[0038] Figure 9 This is an exemplary view illustrating a method of outputting questions about a user's voice using an electronic device based on an operating application according to an embodiment of the present disclosure;

[0039] Figures 10A and 10B are exemplary views illustrating a method of recommending user application operation according to another embodiment of the present disclosure;

[0040] Figures 11A and 11B are exemplary views illustrating various embodiments according to the present disclosure;

[0041] Figure 12 This is an exemplary view illustrating the operation of an electronic device and a server according to embodiments of the present disclosure;

[0042] Figure 13 This is a system diagram illustrating the operation of an electronic device and a server according to embodiments of the present disclosure;

[0043] Figure 14 This is a system diagram illustrating the operation of an electronic device and a server according to another embodiment of the present disclosure; and

[0044] Figure 15 This is a flowchart illustrating a control method for an electronic device according to an embodiment of the present disclosure. Detailed Implementation

[0045] Various embodiments of the present disclosure will be described herein with reference to the accompanying drawings. However, it should be noted that these various embodiments are not intended to limit the scope of the present disclosure to the specific embodiments, but should be interpreted as including all modifications, equivalents, and / or substitutions of the embodiments. In describing the embodiments, the same reference numerals may be used to refer to the same elements.

[0046] Expressions such as “contains,” “may contain,” “includes,” or “may include” used herein may specify the presence of a feature (e.g., an element such as a number, function, operation, or component), but do not exclude the presence of other features.

[0047] In this disclosure, expressions such as “A or B”, “at least one of A and / or B”, or “one or more of A and / or B” can include all possible combinations of the items listed together. For example, “A or B”, “at least one of A and B”, or “at least one of A or B” can refer to all of the following: (1) at least one of A; (2) at least one of B; or (3) both at least one of A and at least one of B.

[0048] The expressions such as “first,” “second,” “1,” “2,” etc., used in this disclosure may modify various elements regardless of their order and / or importance, and may be used only to distinguish one element from another without limiting the corresponding element.

[0049] When an element (e.g., a first element) is indicated as "operably or communicatively connected to" / "operably or communicatively connected to" or "connected to" another element (e.g., a second element), it can be understood that the element is directly connected to / directly connected to the other element, or connected via another element (e.g., a third element). Conversely, when an element (e.g., a first element) is indicated as "directly connected to" / "directly connected to" or "connected to" another element (e.g., a second element), it can be understood that there is no other element (e.g., a third element) between the two elements.

[0050] The expression “configured (or set)…” as used in this disclosure may be used interchangeably with, for example, “suitable for…”, “capable of…”, “designed to…”, “fit for…”, “enable to…”, or “able to…”, depending on the context. The term “configured (or set)…” in hardware terms need not necessarily mean “specifically designed for.” Rather, in some cases, the expression “device configured to…” may mean that the device, together with other devices or components, “can perform…”. For example, the phrase “configured (or set) to perform a subprocessor of A, B, or C” may refer to a dedicated processor (e.g., an embedded processor) for performing the corresponding operation, or a general-purpose processor (e.g., a central processing unit (CPU) or application processor) capable of performing the corresponding operation by running one or more software programs stored in a memory device.

[0051] Electronic devices according to various embodiments of this disclosure may include, for example, but not limited to, at least one of the following: smartphones, tablet PCs, mobile phones, video phones, e-book readers, desktop PCs, laptop PCs, netbooks, workstations, servers, personal digital assistants (PDAs), portable multimedia players (PMPs), MP3 players, medical devices, cameras, or wearable devices. Wearable devices may include at least one of the following: accessory types (e.g., watches, rings, bracelets, anklets, necklaces, glasses, contact lenses, or head-mounted devices (HMDs)), fabric or clothing embedded types (e.g., electronic clothing), skin-attached types (e.g., skin pads or tattoos), or bio-implantable circuitry. In some embodiments, electronic devices may include, for example, but not limited to, at least one of the following: televisions, digital video disc (DVD) players, sound cards, refrigerators, cleaners, ovens, microwave ovens, washing machines, air purifiers, set-top boxes, home automation control panels, security control panels, media boxes (e.g., Samsung HomeSync). TM Apple TV TM or Google TV TM ), game consoles (such as Xbox) TM PlayStation TM Electronic dictionaries, electronic keys, portable cameras, electronic frames, etc.

[0052] In another embodiment, the electronic device may include at least one of the following: various medical devices (e.g., various portable medical measurement devices (e.g., blood glucose measuring devices, heart rate measuring devices, blood pressure measuring devices, temperature measuring devices, etc.)), navigation devices, Global Navigation Satellite Systems (GNSS), Event Data Recorders (EDR), Flight Data Recorders (FDR), vehicle infotainment devices, marine electronic devices (e.g., marine navigation devices, gyrocompasses, etc.), avionics, security devices, vehicle front units, industrial or personal robots, drones, automated teller machines (ATMs) of financial institutions, point-of-sale (POS) devices, or Internet of Things (IoT) devices (e.g., light bulbs, various sensors, sprinkler systems, fire alarms, temperature regulators, streetlights, ovens, fitness equipment, hot water tanks, heaters, boilers, etc.).

[0053] In this disclosure, the term "user" may refer to a person who uses an electronic device or a device that uses an electronic device (e.g., an artificial intelligence electronic device).

[0054] This disclosure will now be described in more detail with reference to the accompanying drawings.

[0055] Figure 1 This is an exemplary view that briefly illustrates the operation of an electronic device according to this disclosure.

[0056] Electronic device 100 can store user application usage, content usage, web content usage, etc. in knowledge database 460, and output a response about the user's question based on the knowledge database 460 based on the user's input question.

[0057] Specifically, the electronic device 100 can determine the domain of the data to be stored based on the user's application usage, content usage, web content usage, etc., and when the domain is determined, it stores data about the user's application usage, content usage, and web content usage in the determined domain.

[0058] For example, electronic device 100 can store the operation sequence of a user's application in knowledge database 460, and if the user executes a corresponding domain that is determined to be associated with the executed application, the application is controlled based on the data stored in the determined domain and according to the stored operation sequence of the application. Alternatively, electronic device 100 can determine a domain (e.g., a movie domain) associated with a movie application based on reserving a movie ticket through a movie application, and recommend sending confirmation of reservation details to another user based on the data stored in the domain. Alternatively, electronic device 100 can store the user's actions of performing content through a specific application in knowledge database 460, and when a user command is input for replaying the corresponding content, determine the domain associated with the corresponding content, obtain the application for replaying the corresponding content based on the data stored in the domain, and replay the corresponding content through the obtained application. Alternatively, electronic device 100 can determine the domain associated with the bookmarked content based on a user command for bookmarking specific input content, and bookmark the content associated with the bookmarked content based on the data stored in the determined domain.

[0059] In other words, electronic device 100 can obtain various data based on its various usage histories and store them in knowledge database 460 by domain, and output a response based on the user's input question and the questions stored in knowledge database 460.

[0060] A knowledge database 460 can refer to a database that stores datasets according to an appropriate architecture, including subject data, object data, and predicate data. That is, a knowledge database 460 can express data and the types of connections between data in a general form (e.g., the tabular form of a relational database) and store the set of expressed data and the types of connections between data. Subject data can refer to data representing the subject to be represented, predicate data can refer to data representing the relationship between the subject and the object, and object data can refer to data representing the content or value of the connection relationship. The set including subject data, connection relationship data, and object data can be called triple data.

[0061] The knowledge database 460 can be established by storing triple data obtained by electronic device 100 through web pages or various applications in the form of relational database tables.

[0062] According to the implementation method, the electronic device 100 can obtain triplet data by performing morpheme analysis and parsing on the identified text on the webpage. For example, if the text "The current population of Seoul is 9.77M" is on the webpage, the electronic device 100 can perform morpheme analysis and parsing on the text and obtain triplet data including "Seoul (subject data), population (predicate data), 9.77M (object data)". The knowledge database 460 can be established by storing the triplet data obtained by the electronic device 100 in a tabular form of a relational database as shown in Table 1.

[0063] Table 1

[0064] Main data Predicate data Object data Seoul population 9.77M

[0065] According to another embodiment, electronic device 100 can obtain triple data including "application name (subject data), performed function (predicate data), and content name (object data)" based on application usage. For example, based on performing a movie reservation in a movie application, electronic device 100 can obtain triple data including "movie application name (subject data), movie reservation (predicate data), and movie title (object data)". Knowledge database 460 can then be established by storing the triple data obtained by electronic device 100 in a tabular form of a relational database as shown in Table 2.

[0066] Table 2

[0067] Main data Predicate data Object data Movie application name Movie reservation Movie title

[0068] According to the implementation, the electronic device 100 can output a response about the user's voice based on triple data stored in the knowledge database 460. Specifically, based on the user's voice including the question being entered, the electronic device 100 can grasp the intent of the question by recognizing and analyzing the user's voice, and determine whether the grasped intent of the question corresponds to any predicate data in the triple data stored in the knowledge database 460. For example, based on the user's voice "What time is a good time to watch the OOO movie this Saturday?" as input, the electronic device 100 can grasp that the user's intent is "movie reservation", and identify the data in the knowledge database 460 where the connection relationship data is "movie reservation". Then, the electronic device 100 can identify the subject data where "movie reservation" in the triple data is predicate data by using the relational data table stored in the knowledge database 460. Then, the electronic device 100 can identify the subject data (movie application) with the highest usage frequency from the identified subject data, and output a response recommending a movie reservation through the identified movie application.

[0069] Figure 2This is a block diagram that briefly illustrates the configuration of an electronic device according to an embodiment of the present invention. Figure 2 As shown, electronic device 100 may include memory 110 and processor 120.

[0070] Memory 110 may store instructions or data associated with at least one other element of electronic device 100. Memory 110 may be implemented as non-volatile memory, volatile memory, flash memory, hard disk drive (HDD), solid-state drive (SSD), etc. Memory 110 may be accessed by processor 120, and processor 120 may perform data read / write / modification / deletion / update operations. In this disclosure, the term "memory" may include memory 110, read-only memory (ROM; not shown) in processor 120, random access memory (RAM; not shown), or a memory card (not shown; e.g., a microSD card, memory stick) installed in electronic device 100. Furthermore, memory 110 may store programs, data, etc., for configuring various screens to be displayed in the display area of ​​display 150.

[0071] Furthermore, memory 110 can store a dialogue system that provides responses to user input (i.e., specifically, user voice). For example... Figure 4 As shown, the dialogue system may include an automatic speech recognizer (ASR) 410, a natural language understanding (NLU) component 420, a dialogue manager (DM) 430, a natural language generator (NLG) 440, a text-to-speech (TTS) component 450, and a knowledge database 460.

[0072] Automatic speech recognizer 410 can perform speech recognition on user speech input via a microphone or the like. Natural language understanding unit 420 can understand the intent of the user's speech based on the speech recognition results. Dialogue manager 430 can obtain information about the response to the user's speech based on the natural language understanding results and data stored in knowledge database 460. For example, dialogue manager 430 can obtain information for generating a response, and as described above, the obtained information can be determined based on the intent of the user's speech grasped by natural language understanding unit 420 and data stored in knowledge database 460. Natural language generator 440 can obtain natural language as a response to the user's speech based on the information obtained by dialogue manager 430. TTS 450 can convert the obtained natural language into speech. Therefore, the dialogue system can provide a response to the user's speech in sound, and the user can perform a dialogue with electronic device 100.

[0073] According to the implementation, the natural language generator 440 can take information obtained through the dialogue manager 430 and the knowledge database 460 as input values ​​for the artificial intelligence model and obtain natural language as a response to the user's voice.

[0074] Knowledge database 460 can store data for personalized responses. The data stored in knowledge database 460 can be modified. According to one embodiment, knowledge database 460 can store at least one of the application type used by electronic device 100, content information related to that application, and functions used through that application. According to another embodiment, knowledge database 460 can store key information including text within the application. According to another embodiment, when a user interaction is detected performing a function related to the application being used, knowledge database 460 can store the performed function and information about the user interaction. According to another embodiment, knowledge database 460 can store category information about the application being executed and knowledge information about the application being executed. According to another embodiment, knowledge database 460 can store information related to continued use of the application, information related to the operation of the application, or information about a specific application matched with information about a specific type of content based on the use of similar types of content when using a specific application, and store such information. Here, the information about the application can be at least one of the application type, content information related to the application being executed, and functions used through the application (payment function, search function, appointment function, etc.).

[0075] Furthermore, memory 110 can store an artificial intelligence (AI) agent for operating the dialogue system. Specifically, electronic device 100 can use the AI ​​agent to generate natural language as a response to user speech. The AI ​​agent, as a dedicated program for providing AI-based services (e.g., speech recognition services, personal assistant services, translation services, search services, etc.), can be executed by a conventional general-purpose processor (e.g., CPU) or a separate AI-specific processor (e.g., GPU, etc.).

[0076] An AI agent can operate based on user-inputted voice. The AI ​​agent obtains responses by feeding user questions into a trained AI learning model.

[0077] If a user's voice is input (i.e., specifically, a trigger voice for executing AI functions), or if a preset button is selected (i.e., a button for executing AI personal assistant functions), the AI ​​agent can operate. Alternatively, the AI ​​agent can be in a pre-execution state before the user's voice is input or the preset button is selected. In this case, the AI ​​agent of electronic device 100 can obtain a response to the user's question after the user's voice is input or the preset button is selected. Furthermore, the AI ​​agent can be in a standby state before the user's voice is input or the preset button is selected. The standby state refers to the state in which predefined user input is detected to control the start of operation of the AI ​​agent. If a user's voice is input or a preset button is selected while the AI ​​agent is in a standby state, electronic device 100 can operate the AI ​​agent and obtain natural language as a response to the user's voice. The AI ​​agent can control various modules described below, which will be detailed below.

[0078] Furthermore, according to an embodiment, memory 110 may store an artificial intelligence model trained to generate (or obtain) responses to user questions. The trained artificial intelligence model according to this disclosure may be built by considering factors such as the application domain of the model and the computer performance of the device. For example, the artificial intelligence model may be trained to obtain natural language using information obtained from dialogue manager 430 and knowledge database 460 as input data. To generate natural language, the trained artificial intelligence model may be based on, for example, a neural network. The artificial intelligence model may be designed to simulate the structure of the human brain and may include multiple network nodes that simulate neurons in a neural network and have weighted values. The multiple network nodes may establish connections to each other, such that the neurons simulate the synaptic activity of neurons that send and receive signals through synapses. Furthermore, the document summarization model may include a neural network model or a deep learning model developed from a neural network model. In a deep learning model, multiple network nodes are located at different depths (or layers) and can send and receive data according to convolutional connections. Examples of trained artificial intelligence models may include, but are not limited to, deep neural networks (DNNs), recurrent neural networks (RNNs), and bidirectional recurrent deep neural networks (BRDNNs).

[0079] Furthermore, in the above embodiment, the artificial intelligence model has been described as being stored in electronic device 100; however, this is merely one implementation, and the artificial intelligence model can be stored in another electronic device. For example, the artificial intelligence model can be stored in at least one or more external servers. Electronic device 100 can receive user voice input and send the received user voice to the external server storing the artificial intelligence model, and the artificial intelligence model stored on the external server can take the user voice received from electronic device 100 as input value and output a result.

[0080] The processor 120 can be electrically connected to the memory 110 and control the overall operation and functions of the electronic device 100.

[0081] Processor 120 can execute applications stored in memory 110. When a user command is entered while executing an application, processor 120 can obtain at least one of the following: the type of the executing application, content information related to the executing application, and functions used through the application, and store this information in memory 110. Specifically, when a user command is entered while executing an application, processor 120 can obtain at least one of the following: the type of the executing application, content information related to the executing application, and functions used through the application, and store this information in knowledge database 460. When a user question related to the application is input to a trained artificial intelligence model, processor 120 can output an answer to the user question, which is determined based on at least one of the type of the executing application, content information related to the executing application, and functions used through the application stored in knowledge database 460.

[0082] The response output by the processor 120 may be at least one of the following: a response providing search results for the user's question, a response forming an action for the user's question, and a response performing an application function for the user's question.

[0083] Based on user questions related to content search, processor 120 can recommend applications for replaying content searched based on information related to the application.

[0084] The processor 120 can obtain a response to a user's question based on at least one of the following: stored domain information, the type of the application being executed, content information related to the application being executed, and the functions used by the application.

[0085] When a user inputs a question, processor 120 can send the question to an external server and receive candidate responses from the external server. Processor 120 can output a response to the user's question based on the received candidate responses and information stored in knowledge database 460. Processor 120 can use an artificial intelligence model to calculate the similarity between the received candidate responses and the data stored in knowledge database 460, and output the candidate responses with the highest similarity to the data in knowledge database 460 from the received candidate responses.

[0086] Although user questions in this disclosure have been described as user voice, the implementation is not limited to this. That is, user questions can be entered in text form.

[0087] The artificial intelligence-related functions disclosed herein can be operated via processor 120 and memory 110. Processor 120 may include one or more processors. The one or more processors may be general-purpose processors such as CPUs, application processors (APs), graphics-specific processors such as GPUs, VPUs, or artificial intelligence-specific processors such as NPUs.

[0088] One or more processors can be configured to process input data according to an artificial intelligence model or predefined operating rules stored in memory 110. The predefined operating rules or the characteristics of the artificial intelligence model can be created through learning.

[0089] Here, "creating through learning" refers to the creation of predefined operating rules for desired characteristics or artificial intelligence models by applying learning algorithms to multiple learning datasets. Learning can be performed within the device itself that performs artificial intelligence according to this disclosure, or it can be performed via a separate server / system.

[0090] Artificial intelligence models may include multiple neural network layers. Each layer may include multiple weights, and the layer's processing is performed by combining the processing results of the previous layer with the processing of the multiple weights. Examples of neural networks may include convolutional neural networks (CNNs), deep neural networks (DNNs), recurrent neural networks (RNNs), restricted Boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRNDDs), deep Q-networks, etc., and implementation methods are not limited to the examples described above, except when explicitly describing a neural network.

[0091] A learning algorithm can be a method of training a predetermined subject machine (e.g., a robot) to determine or predict using multiple learning data. Examples of learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, and implementations are not limited to the examples described above, except where a learning algorithm is explicitly described in this disclosure.

[0092] Figure 3 This is a block diagram illustrating in detail an electronic device according to embodiments of the present disclosure.

[0093] like Figure 3 As shown, in addition to the memory 110 and processor 120, the electronic device 100 may also include a communicator 130, an input device 140, a display 150, and an audio output device 160. However, this embodiment is not limited to the above configuration, and some parts of this configuration may be added or omitted if necessary.

[0094] The communicator 130 may be configured to perform communication with external devices. According to one implementation, the electronic device 100 can receive multiple response candidates regarding a user's voice from an external server via the communicator 130.

[0095] The communicator 130, which communicates with external devices, may communicate via a third device (relay, hub, access point, server, gateway, etc.). Wireless communication may include cellular communication using at least one of, for example, but not limited to, Long Term Evolution (LTE), LTE-A Advanced (LTE-A), Code Division Multiple Access (CDMA), Wideband CDMA (WCDMA), Universal Mobile Telecommunications System (UMTS), Wireless Broadband (WiBro), or Global System for Mobile Communications (GSM). Depending on the implementation, wireless communication may include at least one of, for example, but not limited to, Wi-Fi, Bluetooth, Bluetooth Low Energy (BLE), ZigBee, Near Field Communication (NFC), Magnetic Secure Transmission, Radio Frequency (RF), or Body Area Network (BAN). Wired communication may include at least one of, for example, but not limited to, Universal Serial Bus (USB), High Definition Multimedia Interface (HDMI), RS-232, Power Line Communication (PLC), or Common Old-Style Telephone Service (POTS). The network in which wireless or wired communication is performed may include telecommunications networks, such as at least one of computer networks (e.g., local area networks (LANs) or wide area networks (WANs)), the Internet, or telephone networks.

[0096] The input device 140 may be configured to receive input from user commands. The input device 140 may include a camera 141, a microphone 142, and a touch panel 143.

[0097] Camera 141 may be configured to acquire image data of the surrounding environment of electronic device 100. Camera 141 may capture still or moving images. For example, camera 141 may include one or more image sensors (e.g., front or rear sensors), lenses, image signal processors (ISPs), or flashes (e.g., LEDs, xenon lamps, etc.). Microphone 142 may be configured to acquire sound around electronic device 100. Microphone 142 may receive input external sound signals and generate electronic speech information. Microphone 142 may use various noise cancellation algorithms to eliminate noise generated during the reception of external sound signals. Image or speech information input through camera 141 or microphone 142 may be used as input values ​​for artificial intelligence models.

[0098] The touch panel 143 can be configured to receive various user inputs. The touch panel 143 can receive data through user operation. The touch panel 143 can be configured to connect to a display, which will be described below.

[0099] In addition to the aforementioned camera 141, microphone 142, and touch panel 143, the input device 140 may have various configurations for receiving various types of data.

[0100] The display 150 can be configured to output various images. The display 150 for providing various images can be implemented as a display panel of various forms. For example, the display panel can be implemented using various display technologies, such as, but not limited to, liquid crystal displays (LCDs), organic light-emitting diodes (OLEDs), active-matrix organic light-emitting diodes (AM-OLEDs), liquid crystal on silicon (LCOS), digital light processing (DLP), etc. Furthermore, the display 150 can be in the form of a flexible display and can be connected to at least one of the front surface area, side surface area, rear surface area, etc., of the electronic device 100.

[0101] The audio output device 160 can be configured to output not only various audio data to which an audio processor has performed various processing operations such as decoding, amplification, and noise filtering, but also various notification sounds or voice messages. The audio processor can be a component that performs processing on the audio data. In the audio processor, various processing operations such as decoding, amplification, and noise filtering can be performed on the audio data. The audio data processed in the audio processor can be output to the audio output device 160. The audio output device can be implemented as a speaker, but this is only one implementation method, and it can be implemented as an output terminal capable of outputting audio data.

[0102] As described above, processor 120 can control the overall operation of electronic device 100. Processor 120 may include RAM 121, ROM 122, main central processing unit (CPU) 124, graphics processor 123, first interface 125-1 to nth interface 125-n, and bus 126. RAM 121, ROM 122, main CPU 123, graphics processor 124, first interface 125-1 to nth interface 125-n, etc., can be interconnected via bus 126.

[0103] The ROM 122 can store instruction sets and other data used for system reservation. When a power-on command is input and power is supplied, the main CPU 123 can copy the operating system (O / S) stored in memory to RAM 121 based on the instructions stored in the ROM 122, execute the O / S, and start the system. Once startup is complete, the main CPU 123 can copy various application programs stored in memory to RAM 121 and execute the copied applications to perform various operations.

[0104] The main CPU 123 can access the first memory 110 or the second memory 120, and use the O / S stored in the first memory 110 or the second memory 120 to perform startup. The main CPU 123 can use various programs, contents, data, etc. stored in the first memory 110 or the second memory 120 to perform various operations.

[0105] Interfaces 125-1 through 125-n can be connected to the various components mentioned above. One of the interfaces can become a network interface for connecting to external devices via a network.

[0106] Various embodiments of this disclosure will now be described with reference to Figures 5A to 11B.

[0107] Figures 5A and 5B are exemplary views illustrating data stored in a knowledge database according to embodiments of the present disclosure.

[0108] As shown in Figure 5A, based on the input of a preset user voice, electronic device 100 can store the application or web content-related data that is executing when the preset user command is input in knowledge database 460. For example, based on the input of a user command 510 such as "Bixby! Remember!", electronic device 100 can store the application or web content-related data that is executing when the preset user voice is input in knowledge database 460. The preset user voice input time refers to a preset time before or after the point in time when the user voice is input.

[0109] As shown in Figure 5B, when an operation of pressing button 520 in electronic device 100 is detected, electronic device 100 can store the application or web content-related data that is executing at the time of inputting a preset user command in knowledge database 460. The method of pressing button 520 in electronic device 100 can be changed. For example, electronic device 100 can store the application or web content-related data that is executing at the time of detection of the operation based on an operation of pressing button 520 for a preset time (e.g., 2 seconds) or an operation of pressing button 520 a preset number of times (e.g., 3 times).

[0110] However, this implementation is not limited to the embodiments of Figures 5A and 5B, and the operations used to store data in the knowledge database 460 can be changed. For example, the electronic device 100 may store data related to a running application or web content in the knowledge database 460 based on the application being executed or web content that meets specific conditions. These specific conditions may be, for example, conditions for using an application or web content a predetermined number of times. Alternatively, the specific conditions may be conditions for using an application or web content for a preset time. Alternatively, the specific conditions may be conditions for using all applications or web content detected by the electronic device 100. Alternatively, the specific conditions may be conditions related to touch input that bookmarks specific web content.

[0111] Figure 6 This is an exemplary view illustrating a method for classifying domains (or categories) of data stored in a knowledge database according to embodiments of this disclosure.

[0112] According to the above implementation, when data related to an application or web content to be stored in the knowledge database 460 is determined, the electronic device 100 can determine the domain (or category) of the determined data.

[0113] Specifically, electronic device 100 can determine the domain based on the content of the application being executed or web content.

[0114] For example, such as Figure 6 As shown, based on the execution of a news article in a browser application by electronic device 100, electronic device 100 can determine the domain based on the content of the news article being executed. The determined domain can be, for example, an IT domain.

[0115] Alternatively, based on the game application running on electronic device 100, electronic device 100 can determine the domain based on the game application being run. The domain determined in this case could be, for example, an entertainment domain.

[0116] Alternatively, based on the application executed by electronic device 100 related to payment details, electronic device 100 can determine the domain based on the payment details. For example, if there are multiple frequently visited restaurants in the payment details, the determined domain could be the food domain.

[0117] Domains can be stored in the knowledge database 460. If a new domain needs to be added, the electronic device 100 can add a new domain corresponding to data related to the application or web content. The domain to be added can be one of the domains stored on an external server. Domains can be added or deleted using user commands.

[0118] Based on the fact that no data needs to be added to the knowledge database 460 within a preset time period, the electronic device 100 can delete the corresponding domain. In other words, the electronic device 100 can delete unused domains stored in memory.

[0119] Electronic device 100 can edit fields according to the order in which they are stored. That is, electronic device 100 can determine the priority order of fields based on the size or quantity of data stored in each field. For example, ... Figure 6 As shown, based on the size or quantity of data stored in the domains, in the order of food domain, movie domain, IT domain, and entertainment domain, electronic device 100 can assign priority according to the order of food domain, movie domain, IT domain, and entertainment domain. When a user's voice is input, electronic device 100 can output a response to the user's voice according to the priority order.

[0120] Figure 7 This is an exemplary view illustrating a domain management method according to an embodiment of the present disclosure.

[0121] According to Figure 5A to Figure 6 In the various embodiments shown, electronic device 100 can store various data in knowledge database 460. Based on the increase in data stored in knowledge database 460, electronic device 100 can remove a portion from the stored data and secure storage space. For example, electronic device 100 can first delete the data that was stored most frequently. Alternatively, electronic device 100 can first delete the data that was used least often.

[0122] Electronic device 100 can not only remove a portion of the data, but also send the data to an external server. In other words, electronic device 100 can send old data (or unused data) to an external server, such as a personal cloud server or the secure storage space of electronic device 100.

[0123] Figure 8 This is an exemplary view illustrating a method for obtaining data stored in a knowledge database according to embodiments of the present disclosure.

[0124] Based on the applications currently running, the electronic device 100 can obtain the name of the application, the name of the content being executed within the application, and the function being performed within the application, and store them in the knowledge database 460.

[0125] Electronic device 100 can convert data recognized in applications or web content into text and perform tokenization. Tokenization refers to the process of classifying text based on words. When tokenization is complete, electronic device 100 can perform partial speech annotation (POS). POS refers to the process of recognizing the phonetic components of words in the text and adding tags. Electronic device 100 performs text parsing and can remove preset stop words. After removing stop words, electronic device 100 can determine the headword (i.e., lemma reconstruction). Determining the headword can refer to the process of grouping words by their endings, so that words can be analyzed into individual items identified by word headings or dictionary forms. Through the above process, electronic device 100 can obtain the name of the application, the name of the content, and the name of the function, and store them in knowledge database 460.

[0126] For example, electronic device 100 can execute news articles on a webpage. Electronic device 100 can obtain the original sentence of the corresponding news article (e.g., Samsung R&D has an AI center). The electronic device can separate the original sentence using tokenization, POS, etc. For example, the original sentence can be categorized as "(Samsung R&D) - noun phrase, (has) - verb, (AI center) - noun phrase". Electronic device 100 can generate tuples from the separated sentences, summarize the generated tuples, and store the tuple structure in knowledge database 460. For example, based on the generated tuples being "Samsung R&D, AI center, has", electronic device 100 can store data such as "Samsung R&D, AI center, has", "Samsung R&D, AI center, includes", and "Samsung R&D, AI center, includes" in knowledge database 460.

[0127] As described above, data stored in the knowledge database 460 can be generated under specific circumstances. For example, data stored in the knowledge database 460 can be generated when any of the following conditions are met: no user command is input to the electronic device, the electronic device is charging, or the condition is an early time (e.g., between 00:00 AM and 06:00 PM). In other words, data stored in the knowledge database 460 can be generated even when the user is not using the electronic device 100 but can effectively use it.

[0128] Based on the user's input question while the data is stored in the knowledge database 460, the following will be explained using Figures 5A to 59. Figure 8 The implementation method is described to output a response to a user's question.

[0129] Based on the input user question, electronic device 100 can grasp the user's intent. Specifically, electronic device 100 can grasp the user's intent regarding speech through natural language understanding component 420. Electronic device 100 can determine the appropriate domain based on the intent regarding the user's speech and the domains stored in knowledge database 460. Electronic device 100 can determine the similarity between the grasped user intent and multiple domains stored in knowledge database 460, and determine the domain most similar to the user intent. Electronic device 100 can output a result value regarding the user intent based on data included in the determined domain and the user's intent regarding speech. Electronic device 100 can determine the similarity between the grasped user intent and multiple data included in the determined domain, and determine the data most similar to the user intent. Electronic device 100 can output an answer to the user question based on the determined data.

[0130] Figures 9 to 1 1B is an exemplary view illustrating various embodiments according to this disclosure.

[0131] Figure 9 This is an exemplary view illustrating a method by which an electronic device based on an operating application outputs questions about a user's voice according to an embodiment of this disclosure.

[0132] As described above, electronic device 100 can store data or web content obtained from using various applications in knowledge database 460. That is, electronic device 100 can determine the domains and data stored in knowledge database 460 based on applications or web content. For example, when electronic device 100 executes a news article from a web browser, it analyzes the news article to determine the domain as an IT domain, obtains data such as "Samsung Electronics, KRW 00, operating revenue," and stores this data in knowledge database 460. According to another embodiment, when electronic device 100 executes a movie reservation application, it can determine the domain as a movie domain based on the movie application and store data such as "movie title, movie application name, reservation." Alternatively, electronic device 100 can store data such as "cinema, cinema location, cinema name" based on the movie application. Alternatively, electronic device 100 can store data such as "movie tickets, number of tickets" based on the movie application. According to another embodiment, when electronic device 100 executes an application related to payment details, it can analyze the payment details and store "restaurant name, price, function (payment, cancellation, etc.)."

[0133] When data is stored in the knowledge database 460, based on the voice input "What time is best to see the OOO movie this Saturday?", the electronic device 100 can output an appropriate response based on the user's voice. For example, the electronic device 100 can check, based on information obtained from a calendar application, that the user has no schedule after 2:00 PM on Saturday, and based on information stored in the knowledge database 460, check that the user usually uses the OOO application to make movie reservations, and output a response such as "2:30 PM at the OOO cinema is good. Would you like to reserve two tickets using the OOO application?" The electronic device 100 can obtain information related to the user's preferred seats, the number of movie tickets to be reserved, information related to the location of the cinema, and reserve an appropriate cinema and seat for the user.

[0134] Figures 10A and 10B are exemplary views illustrating a method of recommending user application operation according to another embodiment of the present disclosure.

[0135] As shown in Figure 10A, electronic device 100 can receive user commands (e.g., “Bixby! Remember!”) to store a series of operations, such as booking a movie through a movie application and sharing the movie booking with another user. Specifically, electronic device 100 can store a series of operations, such as capturing the booking screen after booking a movie through a movie application and sharing it with another user through a chat application, in knowledge database 460. Then, as shown in Figure 10B, when the booking screen is executed through the movie application, electronic device 100 can recommend sharing the corresponding booking screen with another user through a chat application. That is, electronic device 100 can not only store data related to one application or web content in knowledge database 460, but also store data related to multiple applications or multiple web content in knowledge database 460. In other words, electronic device 100 can not only store data included in the application or web content itself in knowledge database 460, but also store multiple user commands (e.g., booking a movie, sharing a picture, etc.) input into the application or web content in knowledge database 460, and recommend user commands that are not among the multiple user commands based on a portion of the stored input user commands.

[0136] Figures 11A and 11B are exemplary views illustrating various embodiments according to this disclosure.

[0137] As shown in Figure 11A, based on the user command "Play back OOO", electronic device 100 can search for OOO content and related data in knowledge database 460. If at this time, the user has watched up to segment 6 of OOO content and stored the viewing record in knowledge database 460 through a specific application, electronic device 100 can output the response "You have watched episode 6. Play back episode 7 using the OOO application?"

[0138] As shown in Figure 11B, based on the user command "Show pictures of my daughter," the electronic device can output a response such as "This picture was taken yesterday. Send the picture to my wife via the OO application?" That is, the electronic device 100 can not only provide an answer to the user's question (searching for pictures of the daughter), but can also recommend additional actions not specified by the user. In this case, data related to the series of actions the user performs—searching for pictures of their daughter and then sending them to their wife via the OO application—can be stored in the knowledge database 460.

[0139] As described above, the electronic device 100 can perform the task of understanding the user's intent corresponding to the user's voice and generating natural language corresponding to the resulting data about the user's intent; however, this implementation is not limited to this. That is, as... Figure 12 As shown, server 200 can perform the task of understanding the user's intent corresponding to the user's voice and generating natural language corresponding to the resulting data about the user's intent. In other words, based on the input user voice, electronic device 100 can send the user voice to server 200, and server 200 can understand the user's intent corresponding to the received user voice, generate natural language corresponding to the resulting data about the user's intent, and send the generated natural language to electronic device 100.

[0140] Figure 13 This is a system diagram illustrating the operation of an electronic device and a server according to embodiments of the present disclosure.

[0141] In the above embodiment, a method for the electronic device 100 to output a response based on the knowledge database 460 has been described, but the embodiment is not limited thereto. That is, the electronic device 100 can receive multiple response candidates corresponding to the user's voice from the server 200, and output a response about the user's voice based on the received multiple response candidates and the data stored in the knowledge database 460.

[0142] First, the electronic device 100 can receive user voice input (S1410). Based on the received user voice input, the electronic device 100 can perform speech recognition on the input user voice (S1420). Specifically, speech recognition can be performed by an automatic speech recognizer 410.

[0143] Electronic device 100 can grasp the user's voice intent based on the speech recognition results (S1430). Electronic device 100 can send the grasped user's voice intent to an external server 200 (S1440).

[0144] Server 200 can obtain multiple response candidates based on the user's voice intent (S1450). Server 200 can obtain multiple response candidates based on a database included in server 200. Server 200 can input multiple data from the database including the user's voice intent and an artificial intelligence model, and obtain data similar to the user's voice intent as response candidates. For example, for movie recommendation based on the user's voice intent, server 200 can obtain response candidates, such as response candidates related to the order of currently playing movies and response candidates related to the order of movies by genre.

[0145] The server can send multiple received response candidates to the electronic device 100 (S1460). The electronic device 100 can determine the final response based on the similarity between the data stored in the knowledge database 460 and the multiple received response candidates (S1470). The electronic device 100 can input the data stored in the knowledge database 460 and the data about the multiple received response candidates into an artificial intelligence model, and determine the domain most similar to the user's voice intent, and determine the data in the determined domain that is similar to the user's voice intent as the final response. For example, based on the response candidates related to the order of currently playing movies and the response candidates related to the order of movies by genre received by the electronic device 100 from the server, the electronic device 100 can determine one of the multiple domains from the received response candidates and recommend movies in the data in the movie domain that are similar to the user's question intent. For example, based on information about booking a movie through a movie application and information related to action movies included in the movie domain of the knowledge database 460, the electronic device 100 can output a response for booking an action movie in the currently playing movies. Alternatively, based on the absence of action movies currently playing, the electronic device 100 can output a response for reserving the movie with the highest number of reservations among the currently playing movies. Alternatively, based on the absence of action movies currently playing, the electronic device 100 can download the movie with the highest number of views in the action movie genre, or output a response recommending its viewing. That is, the electronic device 100 can recommend responses to user questions based on data included in the defined domain and the similarity between multiple response candidates.

[0146] The electronic device 100 can use the natural language generator 440 and TTS 450 to output a natural language response about the user's speech (S1480).

[0147] exist Figure 13 In this implementation, speech recognition has been described as being performed by electronic device 100, but the implementation is not limited to this. For example, electronic device 100 may only include an artificial intelligence model for calculating the similarity between knowledge database 460 and multiple response candidates, as well as the similarity between data in knowledge database 460. In this case, electronic device 100 can send the user's voice to server 200, and server 200 can process speech recognition, dialogue management, natural language generation, etc., and electronic device 100 can determine the final response candidate among multiple response candidates.

[0148] Figure 14 This is a system diagram illustrating the operation of an electronic device and a server according to another embodiment of the present disclosure.

[0149] like Figure 14As shown, server 200 may include a speech recognition system and a knowledge database 460. Server 200 may include a typical external server, or it may include a personal cloud server.

[0150] First, the electronic device 100 can receive user voice (S1510). The electronic device 100 can send the received user voice to the server 200 (S1520). The server 200 can perform speech recognition on the received user voice (S1530). Specifically, speech recognition can be performed through automatic speech recognition.

[0151] Server 200 can determine the user's intent based on the speech recognition results (S1540), and obtain multiple response candidates related to the user's speech intent based on the user's speech intent (S1550). Server 200 can obtain multiple response candidates based on a database included in server 200. For example, server 200 can input the user's speech intent and multiple data included in the database into an artificial intelligence model, and obtain data similar to the user's speech intent as response candidates.

[0152] Server 200 can determine the final response based on the similarity between data stored in the knowledge database and multiple received response candidates (S1560). Server 200 can also input the data stored in the knowledge database and data about multiple received response candidates into an artificial intelligence model to determine data similar to the user's voice intent as the final response.

[0153] Server 200 can use a natural language generator and TTS to obtain a natural language response about the user's speech (S1570), and send the obtained natural language response to electronic device 100 (S1580). Electronic device 100 can output the received natural language response (S1590).

[0154] In this disclosure, user voice has been described as input, and natural language response to the input user voice has been described as output; however, this implementation is not limited thereto. That is, the user question input to the electronic device 100 and the output response may be in text form rather than voice.

[0155] Figure 15 This is a flowchart illustrating a control method for an electronic device according to an embodiment of the present disclosure.

[0156] Electronic device 100 can execute applications. Specifically, based on a user command input to execute the application, electronic device 100 can execute the application corresponding to that user command. Electronic device 100 can not only execute applications, but also web browsers, web content, etc.

[0157] Based on the input user command, the electronic device 100 can obtain information about the currently running application and store it in memory (S1610). Specifically, the electronic device 100 can store the information about the currently running application in a knowledge database 460. The user command can be a voice command (e.g., “Bixby! Remember!”) or a command to press a button set on the electronic device 100 in a specific manner. The information about the currently running application can be information related to the name of the running application, the name of the content being performed by the application, the function being performed by the application, etc. Alternatively, the information about the application can be information related to the content displayed by the application (e.g., a news article). Alternatively, the application-related information can be information related to the operation of the application.

[0158] Based on user questions related to the application as input values, the electronic device 100 can output a response to the user questions determined based on information about the stored application (S1620). Specifically, the electronic device 100 can use the artificial intelligence model to calculate the similarity between the user question and data stored in the knowledge database 460, and output the response related to the data with the highest similarity as the answer to the user question. The electronic device 100 can send user voice to the server 200, and the server 200 can send multiple response candidates based on the user voice to the electronic device 100. In this case, the electronic device 100 can use the artificial intelligence model to calculate the similarity between the multiple response candidates and data stored in the knowledge database 460, and output the response candidate with the highest similarity as the response to the user question.

[0159] As used in this disclosure, the terms "part" or "module" can include units configured as hardware, software, or firmware, and can be used interchangeably with terms such as logic, logic block, component, or circuit. A "part" or "module" can be a portion that is integrally formed, or the smallest unit or part of a portion that performs one or more functions. For example, a module can be configured as an application-specific integrated circuit (ASIC).

[0160] One or more implementations can be implemented using software including instructions stored in a machine-readable storage medium (e.g., a computer). A machine can invoke the instructions stored in the storage medium and, as a device capable of operating according to the invoked instructions, can include an electronic device (e.g., electronic device 100) according to the implementation. Based on the instructions executed by the processor, the processor can perform functions corresponding to the instructions directly or under the processor's control using different elements. Instructions can include code generated by a compiler or executed by an interpreter. The machine-readable storage medium can be provided in the form of a non-transitory storage medium. Herein, "non-transitory" means only that the storage medium is tangible and does not include signals, and does not distinguish whether data is stored permanently or temporarily in the storage medium.

[0161] According to one or more embodiments, a method according to one or more embodiments may be provided in a computer program product. The computer program product can be exchanged as a commodity between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., an optical disc read-only memory (CD-ROM)) or through an application store (e.g., a Playstore). TM Online distribution. In the case of online distribution, at least a portion of the computer program product may be temporarily stored in a storage medium such as the manufacturer's server, the application store's server, or the memory of a relay server, or may be temporarily generated.

[0162] Each of the elements (e.g., modules or programs) according to various embodiments may consist of a single entity or multiple entities, and some of the aforementioned sub-elements may be omitted, or different sub-elements may be further included in various embodiments. Alternatively or additionally, some elements (e.g., modules or programs) may be integrated into one entity to perform the same or similar functions as those performed by each respective element prior to integration. According to various embodiments, operations performed by modules, programs, or other elements may be performed sequentially, in parallel, repeatedly, or in a trial-and-error manner, or at least some operations may be performed in a different order, may be omitted, or may be added.

Claims

1. Control methods for electronic devices, including: Based on user commands entered while running the application, obtain application information including the type of the application, content information related to the running application, and application information used through the application; The application information, including the type of the application, content information related to the running application, and the functions used by the application, is stored in the memory as a knowledge database, wherein the knowledge database includes application information about multiple applications; User questions related to the application are input into a trained artificial intelligence model and analyzed by the artificial intelligence model to obtain application information related to the user questions from the knowledge database. Determine the domain information of the application information based on the application information; Based on the domain information, the application information is categorized and stored in the memory; and In response to an input user question, the user question is fed into a trained artificial intelligence model, which then outputs a response to the input user question. The step of inputting the user question into a trained artificial intelligence model and outputting a response to the input user question includes: Determine the domain information of the user question based on the intent of the user question; Based on the domain information of the determined user question and the domain information of the application information stored in the memory, the target application information for generating a response to the user question is determined from the memory; The answer to the user's question is determined based on the target application information, and The output of the response to the input user question includes: Based on the fact that the user's question is related to content search, an application for playing the searched content is provided based on information related to the application.

2. The method according to claim 1, wherein, The storage includes: The running application includes text, extracts key information from the text, and stores it.

3. The method according to claim 1, wherein, The storage includes: Based on detected user interactions performing functions on the running application, historical information is stored, including information about the functions performed and information about the user interactions.

4. The method according to claim 1, wherein, The output is characterized in that it provides at least one of the following: providing search results for the user's question, performing an action related to the user's question, and performing an application function related to the user's question.

5. The method according to claim 1, wherein, The output includes: Based on the user's input question, the user's question is sent to an external server; Receive candidate responses to the user's question from the external server; and Based on the received response candidates and the information stored in the memory, an answer to the user's question is output.

6. The method according to claim 1, wherein, The user question is characterized by being at least one of the user's voice or text.

7. Electronic devices, including: Memory; as well as The processor is configured to run applications stored in the memory. The processor is configured as follows: Based on user commands entered while running the application, obtain application information including the type of the application, content information related to the running application, and application information of the functions used through the application; The application information, including the type of the application, content information related to the running application, and the functions used by the application, is stored in the memory as a knowledge database, wherein the knowledge database includes application information about multiple applications; User questions related to the application are input into a trained artificial intelligence model and analyzed by the artificial intelligence model to obtain application information related to the user questions from the knowledge database. Determine the domain information of the application information based on the application information; Based on the domain information, the application information is categorized and stored in the memory; and In response to an input user question, the user question is fed into a trained artificial intelligence model, and a response to the input user question is output. The step of inputting the user question into a trained artificial intelligence model and outputting a response to the input user question includes: Based on the intent of the user question, determine the domain information of the user question; based on the determined domain information of the user question and the domain information of application information stored in the memory, determine the target application information for generating a response to the user question from the memory; based on the target application information, determine the response to the user question, and The output of the response to the input user question includes: Based on the fact that the user's question is related to content search, an application for playing the searched content is provided based on information related to the application.

8. The electronic device according to claim 7, wherein, The processor is configured to extract key information from the running application, including text, and store it in the memory.

9. The electronic device according to claim 7, wherein, The processor is configured to store historical information in the memory based on detected user interactions that perform functions on the running application. The historical information includes information about the functions performed and information about the user interactions.

10. The electronic device according to claim 7, wherein, The processor is configured to output at least one of providing search results for the user's question, performing an action related to the user's question, and performing an application function related to the user's question.

11. The electronic device according to claim 7, further comprising: communicator, The processor is configured to: send the user question to an external server via the communicator based on the input user question, receive response candidates for the user question from the external server via the communicator, and output a response to the user question based on the received response candidates and information stored in the memory.

Images