Electronic devices and their control methods

Abstract

An electronic device and control method are provided, including an input interface, a communication interface, a memory including at least one command, and at least one processor, wherein the at least one processor is configured to: control the electronic device and execute the at least one command to perform the following operations: receiving user voice through the input interface; determining whether the user voice is related to a task requiring user confirmation by analyzing the user voice; generating a query for user confirmation when the user voice is determined to be related to a task requiring user confirmation; and executing a task corresponding to the user voice when a user response corresponding to the query is input through the input interface. Embodiments may use an artificial intelligence model learned based on at least one of machine learning, neural network, and deep learning algorithms.

Description

Technical Field

[0001] This disclosure relates to an electronic device and a control method thereof. More specifically, this disclosure relates to an electronic device and a control method thereof that provides an inquiry for user confirmation to perform a task corresponding to a user's voice. Background Technology

[0002] Artificial intelligence (AI) systems have recently been used in various fields. AI systems are machines that perform learning and decision-making, and unlike existing rule-based intelligent systems, they become intelligent on their own. With more frequent use, AI systems have improved recognition rates and can more accurately understand user preferences. Therefore, existing rule-based intelligent systems are gradually being replaced by deep learning-based AI systems.

[0003] This type of artificial intelligence technology typically consists of machine learning (e.g., deep learning) and elemental technologies that use machine learning. Machine learning is an algorithmic technique that classifies / learns features from input data, while elemental technologies are techniques that use machine learning algorithms, such as deep learning, to simulate functions of the human brain, such as recognition and decision-making, and consist of technical fields such as language understanding, visual understanding, reasoning / prediction, knowledge representation, and motion control.

[0004] The following are various fields where artificial intelligence technologies are applied: Language understanding is the technology of recognizing and applying / processing human language / characters, and includes natural language processing, machine translation, dialogue systems, question and answer, speech recognition / synthesis, etc. Visual understanding is the technology of recognizing and processing things such as human vision, and includes object recognition, object tracking, image search, human recognition, scene understanding, spatial understanding, image enhancement, etc. Reasoning / prediction is the technology of making decisions and making logical inferences and predictions, and includes knowledge-based / probability-based inference, optimal prediction, preference-based planning, recommendation, etc. Knowledge representation is the technology of automating and processing human experience information into knowledge data, and includes knowledge construction (data creation / classification), knowledge management (data utilization), etc. Motion control is the technology of controlling the autonomous driving of vehicles and the movement of robots, and includes motion control (navigation, collision, driving), manipulation control (behavior control), etc.

[0005] Recently, electronic devices have also been configured to perform various tasks using user voice. For example, electronic devices can perform various tasks via user voice, such as sending messages, sending emails, making remittances, controlling external devices, purchasing products, and reproducing content.

[0006] Electronic devices can perform various tasks via user voice, but there is a problem that electronic devices can perform user-unintentional tasks via user voice or voice output from sources other than the user. As an example, electronic devices can perform user-unintentional tasks via voice output by the user during a conversation with another user or via voice output from an external device (such as a television (TV)).

[0007] Specifically, when a user's unintentional voice is used to perform a task related to user privacy or security, the electronic device may perform the wrong task through the user's unintentional voice, leading to problems such as privacy violations or security vulnerabilities. Therefore, a method is needed to prevent tasks from being performed through the user's unintentional voice.

[0008] The above information is presented as background information only and is intended to aid in understanding this disclosure. No determination or assertion is made regarding whether any of the above content is applicable as prior art to this disclosure. Summary of the Invention

[0009] Technical issues

[0010] The present disclosure is intended to at least address the aforementioned problems and / or disadvantages, and to provide at least the advantages described below. Therefore, one aspect of the present disclosure is to provide an apparatus and method for providing an inquiry for user confirmation to perform a task corresponding to user voice, and a control method thereof.

[0011] Another aspect of this disclosure is to provide an electronic device and its control method capable of determining whether a task related to a user's voice is a task requiring user confirmation, and generating and providing an inquiry for user confirmation based on the determination result.

[0012] Other aspects will be set forth in part in the description which follows, and in part will be apparent from the description, or may be learned by practicing the embodiments presented.

[0013] Solution to the problem

[0014] According to one aspect of this disclosure, an electronic device is provided. The electronic device includes an input interface, a communication interface, a memory including at least one command instruction, and at least one processor connected to the input interface, the communication interface, and the memory and configured to control the electronic device, wherein the at least one processor executes the at least one command to be configured to: receive user voice through the input interface; determine, by analyzing the user voice, whether the user voice is related to a task requiring user confirmation; when it is determined that the user voice is related to a task requiring user confirmation, generate a query for user confirmation; and when a user response corresponding to the query is input through the input interface, execute the task corresponding to the user voice.

[0015] According to another aspect of this disclosure, a method for controlling an electronic device is provided. The method includes: receiving user voice; analyzing the user voice to determine whether the user voice is related to a task requiring user confirmation; when the user voice is determined to be related to a task requiring user confirmation, generating a query for user confirmation; and when a user response corresponding to the query is input, executing a task corresponding to the user voice.

[0016] Beneficial effects of the present invention

[0017] According to the above embodiments, even when a user's unintentional voice is input into an electronic device, user confirmation for performing a task is performed, enabling the user to protect privacy and enhance security.

[0018] Other aspects, advantages, and distinctive features of this disclosure will become apparent to those skilled in the art from the following detailed description of various embodiments disclosed in conjunction with the accompanying drawings. Attached Figure Description

[0019] The above and / or other aspects, features, and advantages of certain embodiments of this disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings, wherein:

[0020] Figure 1 This is a diagram illustrating the use of an electronic device that performs a task in response to a user's voice, according to an embodiment of the present disclosure.

[0021] Figure 2 This is a block diagram illustrating the components of an electronic device according to an embodiment of the present disclosure;

[0022] Figure 3 This is a block diagram illustrating the detailed components of an electronic device according to embodiments of the present disclosure;

[0023] Figure 4 This is a block diagram illustrating a dialogue system of an artificial intelligence agent system according to an embodiment of the present disclosure;

[0024] Figure 5 This is a flowchart describing a control method for an electronic device that provides an inquiry for user confirmation when the user's voice is related to a task requiring user confirmation, according to embodiments of the present disclosure.

[0025] Figure 6a and Figure 6b This is a diagram illustrating examples of generating queries for user confirmation according to various embodiments of this disclosure;

[0026] Figure 7a and Figure 7b This is a diagram illustrating examples of generating queries for user confirmation according to various embodiments of this disclosure;

[0027] Figure 8 This is a diagram illustrating an example of generating a query for user confirmation according to embodiments of the present disclosure;

[0028] Figure 9a and Figure 9b This is a diagram illustrating examples of generating queries for user confirmation according to various embodiments of this disclosure;

[0029] Figure 10 This is a sequence diagram illustrating examples of queries generated by an artificial intelligence system for user confirmation according to embodiments of the present disclosure;

[0030] Figure 11 This is a sequence diagram illustrating another example of an inquiry for user confirmation generated by an artificial intelligence system according to an embodiment of the present disclosure;

[0031] Figure 12 This is a sequence diagram illustrating another example of an inquiry for user confirmation generated by an artificial intelligence system according to embodiments of the present disclosure; and

[0032] Figure 13 This is a sequence diagram illustrating an example of an electronic device or external server converting user speech into text based on a security score of the user's speech according to embodiments of the present disclosure.

[0033] Throughout the accompanying drawings, the same reference numerals will be understood to refer to the same parts, components, and structures. Detailed Implementation

[0034] The following description, provided with reference to the accompanying drawings, is intended to aid in a full understanding of the various embodiments of this disclosure as defined by the claims and their equivalents. It includes various specific details to aid understanding, but these are to be considered exemplary only. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the various embodiments described herein without departing from the scope and spirit of this disclosure. Furthermore, for clarity and brevity, descriptions of well-known functions and structures may be omitted.

[0035] The terms and words used in the following description and claims are not limited to their literal meaning, but are used only to enable a clear and consistent understanding of this disclosure. Therefore, it will be apparent to those skilled in the art that the following description of various embodiments of this disclosure is for illustrative purposes only and not for limiting the disclosure as defined by the appended claims and their equivalents.

[0036] It should be understood that, unless the context explicitly indicates otherwise, the singular form includes plural reference. Thus, for example, a reference to “component surface” includes a reference to one or more such surfaces.

[0037] In this disclosure, the terms "have", "may have", "include", "may include" indicate the presence of a corresponding feature (e.g., a value, function, operation, component such as a part, etc.) and do not exclude the presence of other features.

[0038] In this disclosure, expressions such as “A or B”, “at least one of A and / or B”, “one or more of A and / or B” can include all possible combinations of the items listed together. For example, “A or B”, “at least one of A and B” or “at least one of A or B” can indicate all of the following: 1) the case that includes at least one A, 2) the case that includes at least one B, or 3) the case that includes both at least one A and at least one B.

[0039] The expressions “first,” “second,” etc., used in this disclosure may refer to various components regardless of the order and / or importance of the components. They will only be used to distinguish one component from other components and do not limit the corresponding components.

[0040] When references are made to any component (e.g., the first component) being (operably or communicatively) combined with or connected to another component (e.g., the second component), it should be understood that any component is directly combined with or can be combined with another component (e.g., the third component). When references are made to any component (e.g., the first component) being "directly combined" or "directly connected" to another component (e.g., the second component), it should be understood that there is no other component (e.g., the third component) between any component and the other component (e.g., the second component).

[0041] Depending on the context, the expression “configured (or set) as” as used in this disclosure may be replaced by expressions such as “suitable for,” “capable of,” “designed for,” “suitable for,” “manufactured as,” or “capable of.” The term “configured (or set) as” may not necessarily mean “specifically designed for” in hardware. Rather, the expression “a device configured as…” may indicate that the device can “perform” together with other devices or components. For example, “a processor configured (or set) to perform A, B, and C” may refer to a dedicated processor (e.g., an embedded processor) for performing the respective operations or a general-purpose processor (e.g., a central processing unit (CPU) or application processor) that can perform the respective operations by executing one or more software programs stored in a memory device.

[0042] Electronic devices according to various embodiments of this disclosure may include at least one of, for example, smartphones, tablet PCs, mobile phones, video phones, e-book readers, desktop PCs, laptop PCs, netbooks, workstations, servers, personal digital assistants (PDAs), portable multimedia players (PMPs), MP3 players, medical devices, cameras, or wearable devices. Wearable devices may include at least one of accessory wearable devices (e.g., watches, rings, bracelets, anklets, necklaces, glasses, contact lenses, or head-mounted devices (HMDs), fabric or clothing integrated wearable devices (e.g., electronic clothing), body-attached wearable devices (e.g., skin pads or tattoos), and body-implanted wearable devices. In some embodiments, electronic devices may include, for example, televisions (TVs), digital video disc (DVD) players, audio players, refrigerators, air conditioners, vacuum cleaners, ovens, microwave ovens, washing machines, air purifiers, set-top boxes, home automation control panels, security control panels, and media boxes (e.g., Samsung Electronics' HomeSync). TM Apple's TV TM Or Google TV TM ), game consoles (e.g., Xbox) TM PlayStation TM ( ), at least one of the following: electronic dictionary, electronic key, camera, or digital photo frame.

[0043] In other embodiments, the electronic device may include at least one of various medical devices (e.g., various portable medical measurement devices such as blood glucose meters, heart rate monitors, blood pressure monitors, thermometers, etc.), magnetic resonance angiography (MRA), magnetic resonance imaging (MRI), computed tomography (CT), imaging devices, ultrasound devices, etc.), navigation devices, global navigation satellite systems (GNSS), event data loggers (EDR), flight data loggers (FDR), automotive infotainment devices, marine electronic devices (e.g., marine navigation devices, gyrocompasses, etc.), avionics, safety devices, automotive head units, industrial or household robots, drones, automated teller machines (ATMs) of financial institutions, point-of-sale (POS) devices in stores, or Internet of Things (IoT) devices (e.g., light bulbs, various sensors, sprinkler systems, fire alarms, thermostats, streetlights, toasters, fitness equipment, hot water tanks, heaters, boilers, etc.).

[0044] In this disclosure, the term "user" may refer to a person using an electronic device or a device using an electronic device (e.g., an artificial intelligence electronic device).

[0045] This disclosure will be described in detail below with reference to the accompanying drawings.

[0046] Figure 1 This is a diagram illustrating the use of an electronic device that performs tasks based on a user's voice, according to an embodiment of the present disclosure.

[0047] Reference Figure 1 The electronic device 100 can receive a trigger voice for activating an artificial intelligence agent. As an example, the trigger voice may include trigger words such as "Bixby," "Siri," etc. In this case, the artificial intelligence agent may include a dialogue system that can process or provide responses to user voice as natural language and perform tasks based on user voice. In this case, in addition to the trigger words for activating the artificial intelligence agent, the electronic device 100 may also receive user voice after a specific button set in the electronic device 100 is selected. The electronic device 100 can activate the artificial intelligence agent in response to the trigger voice.

[0048] The electronic device 100 can receive user voice after the artificial intelligence agent program is activated. In this case, the user voice can be voice used to perform a specific task.

[0049] As an example, such as in Figure 1 As shown, the electronic device 100 can receive the user's voice 10 "send a text message to XX saying I will be late today".

[0050] The electronic device 100 can analyze user speech to determine (recognize) whether the user speech is related to a task requiring user confirmation. Specifically, the electronic device 100 can obtain text of the user speech 10 through an automatic speech recognition module, and determine (recognize) whether the user speech is related to a task requiring user confirmation based on the obtained text.

[0051] As an example, the electronic device 100 can determine (identify) whether the acquired user voice is for performing a predetermined task related to user privacy (e.g., remittance task, product purchase task, email sending task, message sending task, telephone call task, etc.) or requires an authentication process. In other words, the electronic device 100 can determine (identify) whether the user voice corresponds to a task related to user privacy or requiring an authentication process, thereby determining whether the user voice is related to a task requiring user confirmation.

[0052] As another example, electronic device 100 can identify tasks related to user speech and entities for performing those tasks based on text obtained from user speech. Furthermore, electronic device 100 can obtain a security score for the user speech based on the identified tasks and entities, and determine whether the user speech is related to a task requiring user confirmation based on the security score. Here, if the security score is a predetermined value or greater, electronic device 100 can determine that the user speech is related to a task requiring user confirmation.

[0053] When it is determined that the user's voice is related to a task requiring user confirmation, the electronic device 100 can generate (obtain) a query for user confirmation. In this case, the electronic device 100 can generate (obtain) a query unrelated to the user's voice. As an example, as in Figure 1 As shown, electronic device 100 can generate (obtain) a query 20 that is completely unrelated to the message sending task: “If you want to send a message, tell me today’s weather.”

[0054] As an example, the electronic device 100 can generate (obtain) a query for user confirmation based on a security score. Specifically, as the security score increases, the electronic device 100 can generate (obtain) a query that is low in relevance to the task corresponding to the user's voice. That is, the electronic device 100 can generate (obtain) a query that is not relevant to the current query. Conversely, as the security score decreases, the electronic device 100 can generate (obtain) a query that is highly relevant to the task corresponding to the user's voice. As an example, the electronic device 100 can extract at least one text related to the task from the text included in the user's voice and generate (obtain) a query to induce user utterances in response to the extracted at least one text.

[0055] Additionally, when the security score is a threshold or higher, the electronic device 100 can provide authentication messages for user authentication and user confirmation. As an example, the electronic device 100 can provide authentication messages for user authentication based on pre-registered user information. In this case, the authentication message may include a query requesting at least one piece of information. For example, the authentication message may include a query requesting user information for authentication, and may also include a query requesting at least one additional piece of information besides the user information for authentication. As an example, the electronic device 100 can provide authentication messages that include a query requesting specific words containing user information for the purpose of user authentication, and in addition to queries requesting specific words containing user information, it may also provide authentication messages that include a request to drag a specific shape (e.g., a straight line, a star, etc.) on the display or a request for biometric authentication (e.g., fingerprint, iris, etc.).

[0056] As another example, electronic device 100 can analyze the voice of a user who regularly uses electronic device 100 to obtain voice feature information about text that can be distinguished from the voice feature information of another user, and store the voice feature information about the text in a memory. Additionally, when it is determined that the user's voice is related to a task requiring user confirmation, electronic device 100 can generate a query including the stored text for user authentication purposes.

[0057] According to another embodiment, the electronic device 100 may randomly select and generate one of a plurality of stored queries.

[0058] According to another embodiment, the electronic device 100 can display multiple objects on a display and generate a query requiring selection of one of the multiple objects.

[0059] According to another embodiment, the electronic device 100 can generate an inquiry based on the surrounding environment. Specifically, the electronic device 100 can determine its location (e.g., home, vehicle, public place, etc.) through various methods. As an example, the electronic device 100 can determine its location based on network information connected to the electronic device 100, Global Positioning System (GPS) information, information about external devices connected to the electronic device 100, etc. Optionally, the electronic device 100 can determine its location or the presence of people around it based on external noise acquired through a microphone, environmental noise vectors, the presence of conversations with people nearby, etc. When it is determined that the electronic device 100 is at home or that no one is around it, the electronic device 100 can generate an inquiry that includes personal information. However, when it is determined that the electronic device 100 is outside or that people are around it, the electronic device 100 can generate an inquiry that does not include personal information.

[0060] According to another embodiment, in cases where a security score may not be able to be calculated or the security score calculated in electronic device 100 is within a threshold range, electronic device 100 may calculate a more accurate security score using a learned artificial intelligence model provided in an external server. In this case, the learned artificial intelligence model (i.e., an artificial intelligence model learned based on user voice and security scores input to various electronic devices 100) can calculate a more accurate security score compared to methods that include calculating the security score in electronic device 100.

[0061] When a query is generated, the electronic device 100 can output the generated query 20. In this case, the electronic device 100 can output the generated query in natural language form through the natural language generation module.

[0062] Electronic device 100 can receive a user's response to an inquiry. As an example, such as in... Figure 1 As shown, the electronic device 100 can receive a user response 30 "good".

[0063] Electronic device 100 can determine whether a user response corresponds to a query. For example, if the query requests user information, electronic device 100 can determine whether the user response includes accurate user information; if the query requests specific words, electronic device 100 can determine whether the user response includes specific words; and if the query requests selection of one of several objects displayed on a screen, electronic device 100 can determine whether the requested object has been selected.

[0064] When a user response corresponding to an inquiry is input, the electronic device 100 can perform a task corresponding to the user's voice and provide the result of the performed task. As an example, the electronic device 100 can perform a text sending task as a task corresponding to the user's voice 10 and output a guiding message 40 indicating the result of performing the text sending task, "Send a message to XX".

[0065] According to the above embodiments, the electronic device 100 can perform tasks corresponding to the user's voice through user confirmation, thereby protecting user privacy and enhancing security.

[0066] The above embodiments have described a scenario where the AI ​​agent system is stored in electronic device 100, and electronic device 100 directly determines whether the user's voice is related to a task requiring user confirmation and generates a query; however, this is merely an example. Some of the above operations can be implemented by an external server or other device. As an example, an external server can obtain the text of the user's voice, determine whether the user's voice is related to a task requiring user confirmation, or generate a query.

[0067] As described above, the electronic device 100 may use an artificial intelligence agent to provide responses to the user's voice. In this case, the artificial intelligence agent, as a dedicated program for providing artificial intelligence (AI) based services (e.g., voice recognition services, secretarial services, translation services, search services, etc.), may be run by an existing general-purpose processor (e.g., a central processing unit (CPU)) or a separate AI-specific processor (e.g., a graphics processing unit (GPU), etc.). Specifically, the artificial intelligence agent may control various modules (e.g., a dialogue system) as described below.

[0068] Specifically, the AI ​​agent can be operated when a predetermined user voice (e.g., "Bixby") is input or a button set in the electronic device 100 (e.g., a button for running the AI ​​agent) is pressed. Furthermore, the AI ​​agent can analyze the user voice to determine whether it is related to a task requiring user confirmation, and generate and provide queries based on the determination result.

[0069] The AI ​​agent can be operated when a predetermined user voice (e.g., "Bixby," etc.) is input or a button (e.g., a button for running the AI ​​agent) on the electronic device 100 is pressed. Alternatively, the AI ​​agent can be in a running state before the user voice is input or the button (e.g., the button for running the AI ​​agent) on the electronic device 100 is pressed. In this case, after the predetermined user voice (e.g., "Bixby," etc.) is input or the button (e.g., the button for running the AI ​​agent) on the electronic device 100 is pressed, the AI ​​agent of the electronic device 100 can perform tasks related to the user voice. For example, if the AI ​​is run by a dedicated AI processor, the functions of the electronic device 100 can be run by a general-purpose processor before the predetermined user voice (e.g., "Bixby," etc.) is input or the button (e.g., the button for running the AI ​​agent) on the electronic device 100 is pressed, and the functions of the electronic device 100 can be run by the dedicated AI processor after the predetermined user voice (e.g., "Bixby," etc.) is input or the button (e.g., the button for running the AI ​​agent) on the electronic device 100 is pressed.

[0070] Additionally, the AI ​​agent can be in a standby state before user voice input is received or a button (e.g., a button for running the AI ​​agent) is pressed on the electronic device 100. Here, the standby state is the state in which predefined user input is sensed and received to control the start of operation of the AI ​​agent. When a predetermined user voice (e.g., "Bixby") is input or a button (e.g., a button for running the AI ​​agent) is pressed while the AI ​​agent is in standby state, the electronic device 100 can run the AI ​​agent and use the running AI agent to perform the task based on the user voice.

[0071] Additionally, the AI ​​agent may be in a terminated state before user voice input (e.g., "Bixby," etc.) or button provided in electronic device 100 (e.g., a button for running the AI ​​agent) is pressed. When a predetermined user voice input (e.g., "Bixby," etc.) is made while the AI ​​agent is in a terminated state, or when a button provided in electronic device 100 (e.g., a button for running the AI ​​agent) is pressed, electronic device 100 may run the AI ​​agent and use the running AI agent to perform tasks based on the user voice input.

[0072] The artificial intelligence agent can control various devices or modules described below. These will be described in detail below. Additionally, specific examples will be described below, using various embodiments, of analyzing user speech using various learning models between electronic device 100 and a server to determine whether the user speech is relevant to a task requiring user confirmation, and generating and providing queries based on the determination results.

[0073] Figure 2 This is a block diagram illustrating components of an electronic device according to an embodiment of the present disclosure.

[0074] Reference Figure 2 The electronic device 100 may include an input interface 110, a communication interface 120, a memory 130, and at least one processor 140. However, the electronic device is not limited to including the above-described components, and some components may be added, omitted, or combined depending on the desired type of electronic device.

[0075] Input interface 110 can receive user input for controlling electronic device 100. As an example, input interface 110 can receive various user operations for controlling electronic device 100, such as user touch, user voice, etc. Specifically, input interface 110 can receive user voice for performing tasks.

[0076] Communication interface 120 is a component for performing communication with any number of external devices and can perform various communications with external electronic devices as needed. Communication connections between communication interface 120 and external devices may include communication via a third device (e.g., a repeater, hub, access point, server, gateway, etc.). Wireless communication may include cellular communication using at least one of, for example, Long Term Evolution (LTE), LTE-A Advanced (LTE-A), Code Division Multiple Access (CDMA), Wideband CDMA (WCDMA), Universal Mobile Telecommunications System (UMTS), Wireless Broadband (WiBro), or Global System for Mobile Communications (GSM). According to embodiments, wireless communication may include at least one of, for example, Wireless Fidelity (WiFi), Bluetooth, Bluetooth Low Energy (BLE), Zigbee, Near Field Communication (NFC), Magnetic Secure Transmission, Radio Frequency (RF), or Body Area Network (BAN). Wired communication may include at least one of, for example, Universal Serial Bus (USB), High Definition Multimedia Interface (HDMI), RS-232, Power Line Communication, or Common Old-Style Telephone Service (POTS). Networks that perform wireless or wired communication may include at least one of telecommunications networks (e.g., computer networks (e.g., local area networks (LANs) or wide area networks (WANs)), the Internet, or telephone networks).

[0077] Additionally, the communication interface 120 can perform communication with an external server used to provide artificial intelligence services. Specifically, the communication interface 120 can send user voice to the external server and receive queries from the external server for user confirmation.

[0078] Memory 130 may store commands or data associated with at least one other component of electronic device 100. Specifically, memory 130 may be implemented using non-volatile memory, volatile memory, flash memory, hard disk drive (HDD), solid-state drive (SDD), etc. Memory 130 may be accessed by processor 140, and the reading, recording, correction, deletion, updating, etc., of data in memory 130 may be performed by processor 140. In this disclosure, the term "memory" may include memory 130, read-only memory (ROM) (not shown) in processor 140, random access memory (RAM) (not shown), or a memory card (not shown) (e.g., a Micro Secure Digital (SD) card or Memory Stick) installed in electronic device 100. Additionally, programs, data, etc., used to configure various screens to be displayed on the display area of ​​the monitor may be stored in memory 130.

[0079] Furthermore, memory 130 may store an artificial intelligence agent for operating the dialogue system. Specifically, electronic device 100 may use the artificial intelligence agent to generate natural language as a response to user speech. In this case, the artificial intelligence agent may be a dedicated program for providing AI-based services (e.g., speech recognition services, secretarial services, translation services, search services, etc.). Specifically, the artificial intelligence agent may be executed by an existing general-purpose processor (e.g., CPU) or a separate AI-specific processor (e.g., GPU, etc.).

[0080] Additionally, the memory 130 may include configurations such as in Figure 4 The diagram illustrates multiple components (or modules) of the dialogue system. Specifically, memory 130 may include a command classification module 430 and a query generation module 440. (Refer to...) Figure 4 This will be described in detail.

[0081] Processor 140 is electrically connected to memory 130 to control the general operation and functions of electronic device 100. Specifically, processor 140 can execute at least one command stored in memory 130 to receive and analyze user voice through input interface 110 to determine whether the user voice is related to a task requiring user confirmation. Furthermore, when it is determined that the user voice is related to a task requiring user confirmation, processor 140 can generate and provide a query for user confirmation. Additionally, when a user response corresponding to the query is input through input interface 110, processor 140 can perform a task corresponding to the user voice.

[0082] Specifically, processor 140 can determine whether the task related to the user's voice requires user confirmation, such as a task related to user privacy or security. As an example, processor 140 can determine whether the task related to the user's voice is one of a remittance task, a product purchase task, an email sending task, a message sending task, or a telephone call task.

[0083] Additionally, the processor 140 can analyze the user's voice input through the input interface 110 to identify the task related to the user's voice and the entity used to perform the task, obtain a security score for the user's voice based on the identified task and entity, and determine whether the user's voice is related to a task that requires user confirmation based on the security score.

[0084] Furthermore, when the user's voice is related to a task requiring user confirmation based on a security score, the processor 140 can generate a query based on the security score. As an example, the processor 140 can generate queries with low relevance to the task corresponding to the user's voice as the security score increases, and generate queries with high relevance to the task corresponding to the user's voice as the security score decreases.

[0085] Additionally, when the security score is a threshold or higher, the processor 140 can provide authentication messages for user authentication.

[0086] Furthermore, the processor 140 can extract at least one text from the text included in the user's speech and generate a question to induce user utterances in response to the extracted at least one text.

[0087] Furthermore, the processor 140 can analyze the speech of a user using the electronic device 100 to obtain speech feature information about the text that distinguishes it from the speech feature information of another user, and store the speech feature information about the text in the memory 130. Additionally, when it is determined that the user's speech is related to a task requiring user confirmation, the processor 140 can generate a query including the stored text. Furthermore, when the speech feature information about the text is obtained in the user's response, the electronic device 100 can perform a task corresponding to the user's speech.

[0088] Figure 3 This is a block diagram illustrating in detail the components of an electronic device according to embodiments of the present disclosure.

[0089] Reference Figure 3 The electronic device 100 may include an input interface 110, a communication interface 120, a memory 130, a display 150, a speaker 160, a sensor 170, and a processor 140. (See also...) Figure 2 Described in Figure 3 The input interface 110, communication interface 120, memory 130 and processor 140 shown are omitted from the description.

[0090] Input interface 110 can receive user input for controlling electronic device 100. Specifically, input interface 110 can receive user voice for performing specific tasks. For example, in Figure 3 As shown, the input interface 110 may include a microphone 111 for receiving user voice, a touch panel 113 for receiving user touches using the user's hand, stylus, etc., and buttons 115 for receiving user manipulations. However, in Figure 3 The input interface 110 shown is merely an example and can be implemented by other input devices (e.g., keyboard, mouse, motion input, etc.).

[0091] The display 150 can display various types of information under the control of the processor 140. Specifically, the display 150 can display a user interface (UI) including multiple objects for querying. Additionally, the display 150 can display a message window including a dialogue between the user and an artificial intelligence agent. The display 150 can be implemented using a touchscreen and a touch panel 113 together.

[0092] Speaker 160 is a component that outputs various alarms or voice messages, as well as various audio data for which various processes (such as decoding, amplification, noise filtering, etc.) are performed by an audio processor. Specifically, speaker 160 can output interrogation or instruction messages in response to user speech as voice messages in natural language form. The component for outputting audio can be implemented by a speaker, but this is only an example, and the component for outputting audio can be implemented by an output terminal capable of outputting audio data.

[0093] Sensor 170 can sense various types of status information of electronic device 100. As an example, sensor 170 may include motion sensors (e.g., gyroscope sensors, accelerometers, etc.) that can sense motion information of electronic device 100, and may include sensors that can sense location information (e.g., GPS sensors), sensors that can sense environmental information around electronic device 100 (e.g., temperature sensors, humidity sensors, atmospheric pressure sensors, etc.), and sensors that can detect user information of electronic device 100 (e.g., blood pressure sensors, blood glucose sensors, pulse rate sensors, etc.). Sensor 170 may also include image sensors for capturing images of the exterior of electronic device 100.

[0094] Figure 4 This is a block diagram illustrating a dialogue system of an artificial intelligence agent system according to an embodiment of the present disclosure.

[0095] Reference Figure 4The dialogue system 400 is a component for performing dialogues with a virtual artificial intelligence agent via natural language. According to embodiments of this disclosure, the dialogue system 400 may be stored in the memory 130 of the electronic device 100. However, this is merely an example, and at least one component included in the dialogue system 400 may be included in at least one external server.

[0096] The dialogue system 400 may include an Automatic Speech Recognition (ASR) module 410, a Natural Language Understanding (NLU) module 420, a Command Classification module 430, a Query Generation module 440, a Dialogue Manager (DM) module 450, a Natural Language Generator (NLG) module 460, and a Text-to-Speech (TTS) module 470. The dialogue system 400 may also include a path planner module, an action planner module, etc.

[0097] Automatic Speech Recognition (ASR) module 410 can convert user speech received from electronic device 100 into text data. For example, ASR module 410 may include a speech recognition module. The speech recognition module may include an acoustic model and a language model. For example, the acoustic model may include information related to phonation, and the language model may include information about combinations of unit phoneme information and unit phoneme information. The speech recognition module can use the phonation-related information and the information about unit phoneme information to convert user speech into text data. Information about the acoustic model and language model may be stored, for example, in an Automatic Speech Recognition Database (ASR DB) 415.

[0098] The natural language understanding module 420 can perform syntactic or semantic analysis to grasp the user's intent. Syntactic analysis can divide the user input into syntactic units (e.g., words, phrases, morphemes, etc.) and grasp which syntactic elements the divided units possess. Semantic analysis can be performed using semantic matching, rule matching, formula matching, etc. Therefore, the natural language understanding module 420 can obtain the domains, intents, or entities (or parameters, slots, etc.) required to express intent from the user input.

[0099] The Natural Language Understanding module 420 can use matching rules, which are categorized into domains, intents, and entities required for capturing intents, to determine the tasks and entities that the user intent intends to perform. For example, a domain (e.g., message) may include multiple intents for determining the task (e.g., message sending, message deletion, etc.), and an intent may include multiple entities (e.g., sending target, sending time, sending content, etc.). Multiple rules may include, for example, one or more basic element parameters. The matching rules may be stored in a Natural Language Understanding Database (NLU DB) 425.

[0100] Natural language understanding module 420 can use language features (e.g., grammatical elements) such as morphemes and phrases to extract the meaning of words extracted from user input and match the extracted word meanings with domains and intentions to determine the task the user intends to perform. For example, natural language understanding module 420 can calculate how many words extracted from user input are included in each domain and intention and determine the task the user intends to perform. According to an embodiment, natural language understanding module 420 can use words as the basis for extracting the task the user intends to perform to determine the entity of the user's speech. According to an embodiment, natural language understanding module 420 can use natural language understanding database 425, which stores language features for extracting tasks related to user speech, to determine tasks related to user speech. In this case, personal information, etc., can be included in natural language understanding database 425.

[0101] The natural language understanding module 420 can generate path rules based on tasks and entities related to the user's speech. For example, the natural language understanding module 420 can select an application to run based on the user's input intent and determine the task to be performed in the selected application. The natural language understanding module 420 can generate path rules by determining the entities corresponding to the determined task. According to an embodiment, the path rules generated by the natural language understanding module 420 may include information about the application to run, the task to run in the application, and the entities required to run the task.

[0102] The natural language understanding module 420 can generate one or more path rules based on tasks and entities related to the user's speech. For example, the natural language understanding module 420 can receive a set of path rules corresponding to the electronic device 100 from the path planner module and map the tasks and entities related to the user's speech to the received set of path rules to determine the path rules. In this case, the path rules may include information about the tasks (or operations) used to perform the functions of the application or information about the entities required to run the tasks. In addition, the path rules may include the order of operations of the application. The electronic device 100 can receive the path rules, select an application according to the path rules, and run the tasks included in the path rules in the selected application.

[0103] The natural language understanding module 420 can generate one or more path rules by determining the application to be run, the task to be run within the application, and the entities required to run the task based on the task and entities related to the user's speech. For example, the natural language understanding module 420 can generate path rules by using information from the electronic device 100 to arrange the application to be run and the task to be run within the application in the form of an ontology or graphical model according to the user's speech intent. The generated path rules can be stored in a path rule database, for example, through a path planner module. The generated path rules can be added to the path rule set of the natural language understanding database 425.

[0104] The natural language understanding module 420 can select at least one path rule from the generated plurality of path rules. As an example, the natural language understanding module 420 can select the optimal path rule from the plurality of path rules. As another example, the natural language understanding module 420 can select the plurality of path rules based on only some tasks specified by the user's voice. The natural language understanding module 420 can determine one of the plurality of path rules through additional input from the user.

[0105] The command classification module 430 can determine whether the user's voice is related to a task requiring user confirmation based on the task associated with the user's voice and the entities obtained by the natural language understanding module 420. Specifically, the command classification module 430 can obtain a security score for the task associated with the user's voice and a security score for the entity obtained by the natural language understanding module 420. In this case, the security score of the task and the security score of the entity can be pre-stored in a security score or privacy level DB 435. Alternatively, the security score of the task can be pre-determined based on whether the task requires a security score, the frequency of use of the task, etc. As an example, as shown in Table 1, the security score of the task can be stored in a security score or privacy level DB 435.

[0106] [Table 1]

[0107]

[0108] Additionally, an entity's security score can be determined based on its importance, quantity, and frequency of use for each task. For example, in the case of a remittance task, electronic device 100 can use logs of the amount used during the remittance to determine the security score. For example, an entity including information about amounts primarily used by the user may have a low security score, while an entity including information about amounts not primarily used by the user may have a high security score. That is, because of the use of user log information, different security scores can be generated for the same amount based on the user information of the user using electronic device 100. As another example, in the cases of messaging tasks, email sending tasks, and telephone tasks, different security scores can be generated based on the frequency of contact with other users and information about those other users (e.g., VIPs and friends) obtained from the user's contact information application. Furthermore, even when the task is performed by the same user, different security scores can be generated and stored based on the type or size of additional items (photo files, video files, audio files, or documents). For example, as shown in Table 2, an entity's security score can be stored in DB 435.

[0109] [Table 2]

[0110]

[0111] The safety scores for the task and entities can be stored in the natural language understanding database 425, but this is only an example, and the safety scores for the task and entities can be calculated by the command classification module 430. For example, information about the task and entities can be input into a pre-learned artificial intelligence model so that the safety scores for the task and entities can be obtained. The command classification module 430 can obtain the safety score of the user's speech based on the obtained safety scores for the task and entities. In this case, the command classification module 430 can obtain the safety score of the user's speech through Equation 1.

[0112] User voice security score =

[0113] (Task safety score) w1 + (entity's security score) w 2 … Equation 1

[0114] As an example, w1 and w2 can be implemented with coefficients such as 0.8 and 0.2 respectively, but this is just an example, and different coefficients can be used to calculate the security score of the user's voice based on user information, etc. Furthermore, Equation 1 is just an example, and different equations can be used to calculate the security score of the user's voice based on the type of entity, the type of task, user information, etc.

[0115] Additionally, when the task or entity corresponding to the user's voice cannot be determined, making it impossible to calculate the safety score, or when the safety score calculated by a linear equation such as Equation 1 is determined to be within a threshold range, the command classification module 430 may request an external server to calculate the safety score for a more accurate result. In this case, the external server may use a learned artificial intelligence model with higher accuracy than that of a linear equation to calculate the safety score.

[0116] Additionally, the command classification module 430 can determine whether a task related to the user's voice requires user confirmation based on the obtained security score of the user's voice. Specifically, the command classification module 430 can determine whether a task related to the user's voice requires user confirmation based on whether the security score of the user's voice is a predetermined value (e.g., 2) or greater. As an example, if the task related to the first user's voice is a remittance task and the remittance amount is $200, the command classification module 430 can determine that the security score of the first user's voice is 5. 0.8+2 0.2 = 4.4, and since the safety score of the first user's voice is a predetermined value or greater, it can be determined that the task associated with the first user's voice is a task requiring user confirmation. As an example, if the task associated with the second user's voice is a phone call task and the phone call target is a home, the command classification module 430 can determine that the safety score of the second user's voice is 2. 0.8+1 0.2 = 1.8, and since the security score of the second user's voice is less than the predetermined value, it can be determined that the task related to the second user's voice is not a task that requires user confirmation.

[0117] In the above embodiment, the command classification module 430 determines whether a task related to the user's voice requires user confirmation based on a security score. However, this is only an example, and the command classification module 430 can also determine whether a task related to the user's voice is a predefined task to determine whether it requires user confirmation. For example, if the task related to the user's voice is one of a pre-stored remittance task, product purchase task, email sending task, message sending task, or telephone call task, the command classification module 430 can determine that the task related to the user's voice requires user confirmation.

[0118] When it is determined that a task related to the user's voice requires user confirmation, the command classification module 430 can output information related to the user's voice (e.g., text information, security score information, etc.) to the query generation module 440. When it is determined that a task related to the user's voice does not require user confirmation, the command classification module 430 can output information related to the user's voice (e.g., text information, etc.) to the dialogue manager module 450.

[0119] The query generation module 440 can generate queries for user confirmation to perform a user voice-related task. In this case, the query generation module 440 can also generate queries unrelated to the user voice-related task. For example, if the user voice-related task is a remittance task, the query generation module 440 can ask queries unrelated to the remittance task, such as weather inquiries or text-based prompting inquiries.

[0120] Additionally, the query generation module 440 can generate queries to elicit answers with low utterance frequency from the user. As an example, the query generation module 440 generates queries to elicit responses including text that the user does not currently use frequently (e.g., "If you want to send an email now, please tell me the name of the highest mountain on Jeju Island"). As another example, the query generation module 440 can generate queries that require any number or text utterance (e.g., "If you want to send an email now, please read out xxxx (random number)"). As yet another example, the query generation module 440 can generate queries that require the user's personal information (e.g., "If you want to send an email now, please tell me your date of birth"). In this case, utterance frequency can be calculated using user log information or a text corpus.

[0121] Additionally, the query generation module 440 can generate queries based on the user's voice security score. Specifically, the query generation module 440 can determine the complexity of the query based on the user's voice security score. As an example, the query generation module 440 can generate queries requiring complex responses as the security score increases, and can generate queries requiring simple responses (e.g., yes / no) as the security score decreases.

[0122] Specifically, the query generation module 440 can determine the relevance between the user's speech task and the query based on the user's speech safety score. As an example, as the safety score increases, the query generation module 440 can generate queries unrelated to the user's speech task. That is, the query generation module 440 can generate queries that induce utterances semantically distant from the user's speech task (distant from the command domain tree or command vector in the continuous space). As an example, the query generation module 440 can use an artificial intelligence model (such as a skip-gram model that maps text in a vector space) to represent the text in the vector space. Additionally, the query generation module 440 can use the similarity (i.e., cosine similarity) between the vector angles corresponding to the text represented in the vector space or the distance between two vectors to generate queries that induce utterances semantically distant from the user's speech task. As an example, when the distance between two vectors is 90 degrees or the distance between two vectors is large, the electronic device 100 can determine that the text corresponding to the two vectors is unrelated to each other.

[0123] As the security score decreases, the query generation module 440 can generate queries related to the user's voice task. Additionally, the query generation module 440 can generate queries requiring user authentication based on the user's voice security score. For example, if the user's voice security score is a predetermined value or higher, the query generation module 440 can generate a query requiring the user's personal information for authentication.

[0124] In addition, since the query generation module 440 needs to compare the user's response to the query with the correct answer to the query, the query generation module 440 can use log information such as existing user responses to generate queries that induce the expected utterances with high accuracy.

[0125] Additionally, the query generation module 440 can generate queries based on the surrounding environment of the electronic device 100. Specifically, the query generation module 440 can determine the location of the electronic device 100 (e.g., home, vehicle, public place, etc.) through various methods. As an example, the query generation module 440 can determine the location of the electronic device 100 based on network information connected to the electronic device 100, GPS information, information about external devices connected to the electronic device 100, etc. Specifically, the query generation module 440 can determine whether the location of the electronic device 100 is a home or a public place based on network information connected to the electronic device 100 (e.g., Internet Protocol (IP) address), location information detected by GPS, etc. Furthermore, if the number of external devices searched based on information about external devices (e.g., device type, device name, etc.) searched via a communication module (such as Bluetooth) is one, the query generation module 440 can determine that the location of the electronic device 100 is a home; and if the number of anonymous external devices searched based on information about external devices is multiple, the query generation module 440 can determine that the location of the electronic device 100 is a public place.

[0126] Optionally, the query generation module 440 can determine the location of the electronic device 100 or the presence of people around the electronic device 100 based on external noise acquired through the microphone, environmental noise vectors, and the presence of conversations with people in the vicinity. Specifically, the query generation module 440 can determine whether the location of the current noise is a vehicle, library, restroom, or subway by inputting the audio acquired through the microphone into a learned artificial intelligence model or an acoustic model based on a Hidden Markov Model (HMM). Optionally, the query generation module 440 can determine the location of the electronic device 100 by measuring the magnitude (Db) of the audio acquired through the microphone. Additionally, the electronic device 100 can use environmental noise vectors to determine its location. Specifically, the electronic device 100 can generate and store vector transformation models (such as speaker recognition vector transformations) according to the type of noise. The query generation module 440 can use a speech recognition acoustic model to sense noise components other than the speech portion, convert the noise source segments into environmental vectors using a pre-stored vector transformation model, compare the previously generated vectors for each noise component with the environmental vectors to obtain a score (e.g., cosine distance), and then determine the current location or environment of the electronic device 100 based on the obtained scores. Additionally, the query generation module 440 can determine whether there is a conversation with people around or information about the people engaging in the conversation based on pre-registered speaker information, thereby determining the current location or environment of the electronic device 100 or whether there are people around the electronic device 100.

[0127] The query generation module 440 can generate queries based on the determined location of the electronic device 100 or the presence of another person. Specifically, when it is determined that the electronic device 100 is located at home or that no one is around the electronic device 100, the query generation module 440 can generate queries that include personal information. However, when it is determined that the electronic device 100 is located outside or that someone is around the electronic device 100, the query generation module 440 can generate queries that do not include personal information.

[0128] In addition, the query generation module 440 can generate queries through various methods. (Refer to...) Figures 6a to 9b This will be described in detail.

[0129] The dialogue manager module 450 can perform tasks determined by the natural language understanding module 420. That is, the dialogue manager module 450 can perform tasks based on the tasks and entities obtained from the natural language understanding module 420 and generate responses to user speech.

[0130] Additionally, the dialogue manager module 450 can determine whether the user's intent captured by the natural language understanding module 420 is clear. For example, the dialogue manager module 450 can determine whether the user's intent is clear based on whether the entity information is sufficient. The dialogue manager module 450 can determine whether the entities captured in the natural language understanding module 420 are sufficient to perform the task. According to an embodiment, if the user's intent is unclear, the dialogue manager module 450 can execute feedback to request the user to input the required information. For example, the dialogue manager module 450 can execute feedback to request the user to input information about the entity in order to capture the user's intent. Furthermore, the dialogue manager module 450 can generate and output a message to confirm the user's inquiry, which includes text modified by the natural language understanding module 420.

[0131] According to an embodiment, the dialogue manager module 450 may include a content provider module. Where the content provider module can perform operations based on tasks and entities captured by the natural language understanding module 420, the content provider module can generate results for performing tasks corresponding to user input.

[0132] According to another embodiment, the dialogue manager module 450 may use a knowledge base stored in the knowledge database 455 to provide a response to the user's voice. In this case, the knowledge base may be included in the electronic device 100, but this is only an example, and the knowledge base may be included in an external server.

[0133] The Natural Language Generator (NLG) module 460 can convert information output by the query generation module 440 or the dialogue manager module 450 into text form. The converted text information can be in the form of natural language utterances. The converted text information can be, for example, information about additional input, information guiding the completion of an operation corresponding to user input, or information guiding additional user input (e.g., user input feedback). The converted text information can be displayed on the display 150 of the electronic device 100, or can be converted into speech form by the Text-to-Speech (TTS) module 470.

[0134] The text-to-speech (TTS) module 470 can convert information in text form into information in speech form. The TTS module 470 can receive information in text form from the natural language generation module 440, convert the information in text form into information in speech form, and output the information in speech form to a speaker.

[0135] The natural language understanding module 420, command classification module 430, query generation module 440, and dialogue manager module 450 can be implemented as at least one module. As an example, the natural language understanding module 420, command classification module 430, query generation module 440, and dialogue manager module 450 can be implemented as a single module to determine the user's task and entities, determine whether the task related to the user's voice requires user confirmation based on the determined user's task and entities, and generate a query or obtain a response corresponding to the user's voice (e.g., path rules) based on the determination result. As another example, the command classification module 430 and query generation module 440 can be included in the dialogue manager module 450.

[0136] Figure 5 This is a flowchart describing a control method for an electronic device that provides an inquiry for user confirmation when the user's voice is related to a task requiring user confirmation, according to embodiments of the present disclosure.

[0137] Reference Figure 5 During operation S510, the electronic device 100 can receive user voice. In this case, the electronic device 100 can receive user voice through microphone 111, but this is only an example, and the electronic device 100 can also receive user voice from an external device.

[0138] During operation S520, the electronic device 100 can determine whether the user's voice is related to a task requiring user confirmation. Specifically, the electronic device 100 can determine whether the user's voice is related to a task involving user privacy or requiring security based on whether the user's voice is related to such a task.

[0139] When it is determined in operation S520 that the user's voice is related to a task requiring user confirmation, in operation S530, the electronic device 100 can obtain (generate) a query for user confirmation. In this case, the query for user confirmation may be a query unrelated to the task related to the user's voice. The electronic device 100 can generate the query for user confirmation through various methods, which will be referred to below. Figures 6a to 9b This will be described in detail. The electronic device 100 can output a query for user confirmation. In this case, the electronic device 100 can output the query through the speaker 160, but this is only an example, and the electronic device 100 can also output the query through the display 150.

[0140] In operation S540, electronic device 100 can determine whether a user response corresponding to the query has been entered.

[0141] When a user response corresponding to the query is input during operation S540-Y, the electronic device 100 can perform a task corresponding to the user's voice during operation S550. However, when no user response corresponding to the query is input during operation S540, the electronic device 100 may not perform the task corresponding to the user's voice or may output a query for additional confirmation or additional feedback.

[0142] When it is determined in operation S520 that the user's voice is not related to a task that requires user confirmation, in operation S550, the electronic device 100 can immediately execute the task corresponding to the user's voice.

[0143] In the following text, reference will be made to Figures 6a to 9b Describe various examples of generating queries for user confirmation.

[0144] Figure 6a and Figure 6b This is a diagram illustrating examples of generating queries based on security scores according to various embodiments of the present disclosure.

[0145] Reference Figure 6a The electronic device 100 can receive a user's voice 610 "send my wife a message that we will have dinner together today".

[0146] Electronic device 100 can calculate a security score for the received user voice 610. Referring to Tables 1 and 2 and Equation 1, electronic device 100 can determine the security score of the user voice 610 to be 2.8. Therefore, electronic device 100 can determine that the security score of the user voice is a first threshold (e.g., 3) or less to generate a query with low complexity.

[0147] As an example, electronic device 100 may generate query 620 “If you want to send a message, please tell me “Blue Ocean””.

[0148] Additionally, when the user response 630 "Blue Ocean" is received, the electronic device 100 can perform a message sending task as a task related to the user's voice, and output a guiding message 640 "Message sent" as a result of the task execution.

[0149] Reference Figure 6b The electronic device 100 can receive the user's voice 650 "Send one million won to my wife".

[0150] Electronic device 100 can calculate the security score of the received user voice 650. Referring to Tables 1 and 2 and Equation 1, electronic device 100 can determine that the security score of user voice 650 is 4.4. Therefore, electronic device 100 can determine that the security score of user voice is a second threshold (e.g., 4) or greater to generate a query for user authentication. As an example, electronic device 100 can generate query 660, "If you want to make a remittance, please tell me your wife's birthday and the first two digits of your account password," to request the user's personal information. In this case, to prevent the leakage of personal information, electronic device 100 can generate a query that only requests some personal information instead of all personal information.

[0151] Additionally, when the user response 670 "May 15, 46" is received, the electronic device 100 can determine whether the user response 670 is the correct answer to the query 660. When the user response 670 is the correct answer to the query 660, the remittance task, which is a task related to the user's voice, is executed, and a guiding message 680, "One million yuan has been remitted to your wife," is output as the result of the task execution.

[0152] Figure 7a and Figure 7b This is an illustration used to describe examples of using text-generated queries, including some text in a user's voice, according to various embodiments of this disclosure.

[0153] Reference Figure 7a The electronic device 100 can receive the user's voice 710 "send a message to the manager including my current location information, 'I will be late'".

[0154] Electronic device 100 can calculate the security score of the received user voice 710. Referring to Tables 1 and 2 and Equation 1, electronic device 100 can determine that the security score of the user voice 710 is 3. Therefore, electronic device 100 can determine that the security score of the user voice is a third threshold (e.g., 2) or greater, in order to request an inquiry for user confirmation.

[0155] In this scenario, electronic device 100 can generate an inquiry requesting utterances from at least one text among a plurality of texts obtained by automatic speech recognition module 410. In this scenario, electronic device 100 can generate an inquiry requesting utterances from at least one text among the plurality of texts that has been identified as a keyword. As an example, electronic device 100 can generate the inquiry 720 “If you want to send a message to the manager, please tell me ‘including the location’”. In this scenario, electronic device 100 can use the result value obtained through the Natural Language Understanding (NLU) module to generate an inquiry requesting utterances from at least one text among the plurality of texts that has been identified as a keyword. Specifically, when Named Entity Recognition (NER) is performed in Natural Language Understanding processing, electronic device 100 can generate an inquiry requesting utterances from combined texts by combining the results of NER performance with each other. As an example, given the result of NER performance {Recipient: "Manager", Command: "Send Text", Message: "Late", Additional Content: "Location Information"}, electronic device 100 can generate an inquiry requesting utterances from combinations of texts (such as recipients and additional content besides the message). As another example, without using NLU results, electronic device 100 can generate a query by combining at least one text obtained through automatic speech recognition (ASR) results using predetermined rules. For example, electronic device 100 can generate a query requesting the utterances "manager, current location information, message" by analyzing the morphemes or transcribed portions of the text, where the utterances "manager, current location information, message" are nouns in the text obtained based on ASR results. Additionally, electronic device 100 can generate a query requesting the utterance "location information," where the utterance "location information" is text in the text obtained based on ASR results that the user has not previously spoken.

[0156] Additionally, when a user response 730 "including location transmission" is received, the electronic device 100 may obtain the current location information through the sensor 170 in response to the user response 730, execute a message transmission task to send the message along with the obtained location information to the manager, and output a guiding message 740 "Message transmission completed" as a result of the execution of the task.

[0157] The electronic device 100 can also determine the complexity of the inquiry based on the user's voice security score. For example, as the security score increases, the electronic device can generate an inquiry that requires more words (e.g., "manager, late, location, include, send"), and as the security score decreases, the electronic device can generate an inquiry that requires fewer words (e.g., "manager, send").

[0158] Reference Figure 7bThe electronic device 100 can receive a user's voice 750 "send a message to the manager including my current location information, 'I will be late'".

[0159] Electronic device 100 can calculate the security score of the received user voice 750. Referring to Tables 1 and 2 and Equation 1, electronic device 100 can determine that the security score of the user voice 750 is 3. Therefore, electronic device 100 can determine that the security score of the user voice is a third threshold (e.g., 2) or greater, in order to request an inquiry for user confirmation.

[0160] In this scenario, electronic device 100 can generate an inquiry requesting information about an entity obtained through natural language understanding module 420. Specifically, electronic device 100 can generate an inquiry requesting information about a "sending target," where the "sending target" is an entity obtained through natural language understanding module 420. As an example, electronic device 100 can generate inquiry 760, "If you want to send a message to your manager, please tell me your manager's name."

[0161] Additionally, when the user response 770 “Hong Kil-dong” is received, the electronic device 100 can obtain the current location information through the sensor 170 in response to the user response 770, execute a message sending task to send the message along with the obtained location information to the manager, and output a guiding message 780 “Message sending has been completed” as the result of the task execution.

[0162] Figure 8 This is a diagram illustrating a method for generating queries based on the performance of an electronic device according to embodiments of the present disclosure.

[0163] As described above, when the electronic device 100 does not include the display 150, the electronic device 100 can generate an inquiry requiring a user response in the form of voice. However, when the electronic device 100 includes the display 150, the electronic device 100 can display multiple objects on the display 150 and generate an inquiry that induces the selection of one of the multiple objects.

[0164] Reference Figure 8 The electronic device 100 can receive the user's voice 810 "Order three packs of diapers at xx business center".

[0165] The electronic device 100 can calculate a security score for the received user voice 810. The electronic device 100 can determine that the security score of the user voice 810 is a third threshold (e.g., 2) or greater, and then request an inquiry for user confirmation.

[0166] In this scenario, the electronic device 100 may display a UI 830 that includes multiple objects. In this case, the objects included in the UI 830 may include text or images unrelated to the user's speech. As an example, such as in... Figure 8 As shown, the electronic device 100 can display a UI 830 that includes fruit objects such as "pear", "apple" and "pineapple".

[0167] Additionally, the electronic device 100 can generate a query requiring the selection of at least one of the plurality of objects. As an example, the electronic device 100 can generate query 820 "Please select Apple from the displayed UI".

[0168] Additionally, when a user touches the Apple logo, the electronic device 100 can respond to the user touch to perform a product purchase task and output a guiding message 840 as a result of the task execution: "Three packs of diapers have been ordered."

[0169] Even though electronic device 100 includes display 150, electronic device 100 may display the UI on the display only under specific circumstances (e.g., when the user is currently near electronic device 100). As an example, electronic device 100 may display the UI on the display when it is determined that the user is near the electronic device by analyzing images captured by a camera or voice received by a microphone, or when the user is wearing or using electronic device 100.

[0170] In the above embodiment, the query is a request to select at least one of a plurality of objects displayed on the display, but this is merely an example, and the electronic device 100 may generate a query that requests text corresponding to the at least one object displayed on the display 150. As an example, the electronic device 100 may display the object "apple" on the display 150 and generate the query "Please tell me the name of the fruit displayed on the screen." In the above embodiment, the query is output in audio form, but this is merely an example, and the query may be output on the display 150.

[0171] When it is determined that another electronic device is in use by the user, electronic device 100 may use the other electronic device to generate an inquiry for user confirmation. Electronic device 100 may generate the inquiry for user confirmation based on sensor information that can be obtained from the other electronic device, and receive the sensor information obtained from the other electronic device to determine whether the user responds to the inquiry.

[0172] As a method for determining whether there is another electronic device being used by the user, electronic device 100 can sense that another electronic device identified as being owned by the user is connected to electronic device 100, and when electronic device 100 receives device usage information from the other electronic device, it determines that the user is using the other electronic device.

[0173] Electronic device 100 may use user profile information, account information, authentication information, and biometric information obtained from a biosensor from another electronic device to determine whether a user owns the other electronic device. Device usage information may include information indicating whether a user is wearing the other electronic device and information indicating whether a user is using the other electronic device.

[0174] As an example, information indicating whether a user is wearing another electronic device could be the following: the wearable device and / or electronic device 100's attachable / detachable structure is in a secured state after motion is detected and within a reference time period; a biosignal is detected in the wearable device and / or electronic device 100 and within a reference time period; or the wearable device's attachable / detachable structure is in a secured state and a biosignal is detected in the wearable device. The secure state of the wearable device's attachable / detachable structure can be identified using the sensing values ​​of sensors disposed in the attachable / detachable structure. Sensors disposed in the attachable / detachable structure may include, for example, conductive sensors, Hall sensors, magnetic sensors, etc. Motion sensors or accelerometers disposed in the wearable device and / or electronic device 100 can be used to identify motion of the wearable device and / or electronic device 100. The wearable device and / or electronic device 100 may include biosensors for detecting biosignals. Biosensors may include, for example, heart rate sensors, pulse sensors, blood pressure sensors, sweat sensors, body temperature sensors, iris sensors, fingerprint sensors, etc.

[0175] As an example, if the time since user input is detected is within a reference time, if a user command to update the screen is received, or if the screen is on, the electronic device 100 can determine that the corresponding device is being used. As an example, the electronic device 100 can generate a query (or request) such as "Please shake your smartwatch." Additionally, the electronic device 100 can use motion sensors in the smartwatch to receive motion information about the smartwatch's movement to determine if the user responds. As an example, the electronic device 100 can generate a query (or request) such as "Please perform fingerprint authentication on the device you are currently using." Then, if a fingerprint is identified and authenticated from the fingerprint sensor of another electronic device the user is using, the electronic device 100 can receive the corresponding authentication information to determine if the user responds.

[0176] Figure 9a and Figure 9b This is an illustration used to describe examples of learning speech feature information of some text included in a user's speech and generating queries using text with the learned speech feature information according to various embodiments of the present disclosure.

[0177] When a user inputs voice input, the electronic device 100 can use the user's voice to perform speaker recognition. When the speaker recognition confirms that the user is the user of the electronic device 100, the electronic device 100 can collect the user's voice input.

[0178] Furthermore, when a user of electronic device 100 uses the speaker recognition service, electronic device 100 can obtain the similarity between the voices of different users by comparing the voice features of the user's voice with registered voice features for other users' voices. Specifically, electronic device 100 can obtain voice feature information about the user's voice. In this case, electronic device 100 can obtain voice feature information by analyzing the frequency characteristics of each phoneme included in the user's voice. Specifically, electronic device 100 can use feature extraction techniques such as cepstral projection, linear prediction coefficients (LPC), mel-frequency cepstral coefficients (MFCC), filter bank energy, etc., to obtain the frequency characteristics of each phoneme included in the user's voice. In this case, the frequency characteristics of each phoneme can be represented by a voice feature vector, and can be represented by a two-dimensional vector or a multi-dimensional vector.

[0179] Furthermore, the electronic device 100 can obtain similarity by comparing the speech feature information of each phoneme obtained for each of multiple users. Additionally, the electronic device 100 can identify phonemes with high similarity differences in the similarity of the speech feature information for each phoneme to extract speaker feature information. That is, the electronic device 100 can compare the speech feature information for each phoneme among users to exclude highly similar speech that does not reflect the speaker's characteristics, and store low-similarity speech that reflects the speaker's characteristics as speaker feature information. Furthermore, the electronic device 100 can input the speech feature information (i.e., speech feature vector) of each phoneme obtained for each of multiple users into a learned artificial intelligence model (e.g., a deep neural network (DNN) model) to extract a speaker feature vector indicating the speaker's characteristics. In this case, the speaker speech feature vector can be represented as a two-dimensional vector or a multi-dimensional vector with multiple dimensions.

[0180] Furthermore, the electronic device 100 can assign high weights to phonemes with the largest differences in similarity. Additionally, when large differences in similarity scores exist consecutively within the same phoneme, the electronic device 100 can assign high weights to phonemes with large differences in similarity scores.

[0181] Reference Figure 9a If the text “yesterday” included in the user voice 910 “play the drama I watched yesterday” is distinguished from the “yesterday” spoken by another user at the same time the user speaks the user voice 910 (that is, if the similarity score difference is a threshold or greater), the electronic device 100 may store information 915 about the text “yesterday” and information about the speech features of the text in the memory 130.

[0182] When a new user registers, the electronic device 100 can induce the existing user's speech features to speak in text. That is, the electronic device 100 can compare and store the speech features of the existing user and the new user to further enhance the recognition capability of the existing user's speech features.

[0183] Electronic device 100 can generate a query using text spoken by a user of electronic device 100 that has speech features stored in memory 130 and distinguishable from the speech of another user.

[0184] Reference Figure 9b The electronic device 100 can receive the user's voice 920 "Purchase xx product".

[0185] Electronic device 100 can calculate a security score for the received user voice 920. If the security score of the user voice 920 is a third threshold (e.g., 2) or greater, electronic device 100 can ask a query for user confirmation. Specifically, if the security score is a fourth threshold (e.g., 3.5) or higher that requires user authentication, electronic device 100 can generate a query for user authentication.

[0186] Specifically, electronic device 100 can generate a query using text with voice features that distinguish it from another user. For example, electronic device 100 can use the text "yesterday" to generate a query 930 that requests the phrase "the friend you met yesterday" to perform authentication. The voice password is "the friend you met yesterday," where the text "yesterday" is text with voice features stored in memory 130 and distinguishable from another user. In this case, the voice password may include at least one other text as well as text with voice features distinguishable from another user. Furthermore, the query 930, which includes the voice password, can be changed each time a query is generated. As an example, various voice passwords including the text "yesterday" can be generated (such as "the friend you met yesterday," "yesterday's weather," "the food you ate yesterday," etc.). That is, the voice password can be changed each time a query is generated to prevent user authentication from being performed via recorded voice.

[0187] Additionally, upon receiving user response 940 "The friend I met yesterday," electronic device 100 can compare the voice features included in user response 940 for "yesterday" with pre-stored voice features for "yesterday" to perform user authentication. If the similarity between the voice features included in user response 940 for "yesterday" and the pre-stored voice features for "yesterday" is a threshold or greater, electronic device 100 can perform user authentication and perform tasks related to user voice 920. Furthermore, electronic device 100 can output a guiding message 950 "Authenticated" as an authentication result.

[0188] According to various embodiments of the present disclosure as described above, the electronic device 100 may perform user confirmation or user authentication before performing a task corresponding to the user's voice, in order to prevent the execution of a task unintentionally performed by the user.

[0189] Electronic device 100 may use an artificial intelligence agent program stored in electronic device 100 to perform the operations described above, but this is only an example, and electronic device 100 may combine with an external server to perform the operations described above.

[0190] In the following text, reference will be made to Figures 10 to 12 Describe an example of combining an external server to generate a query for user confirmation.

[0191] Figure 10 This is a sequence diagram illustrating examples of queries generated by an artificial intelligence system for user confirmation according to embodiments of the present disclosure. Figure 10 The diagram illustrates an example of how an external server determines whether a task corresponding to a user's voice requires user confirmation.

[0192] Reference Figure 10 The electronic device 100 may receive a trigger voice (S1005). In this case, the trigger voice may include a trigger word for running an artificial intelligence agent program stored in the electronic device 100. According to another embodiment, in order to execute the artificial intelligence agent program for speech recognition, the electronic device 100 may receive a user command for selecting a specific button set in the electronic device 100.

[0193] The electronic device 100 can activate the artificial intelligence agent in response to a triggered voice (S1010). In this case, the artificial intelligence agent can be a virtual secretary used to provide interactive services to the user.

[0194] The electronic device 100 can receive user voice (S1015). In this case, the user voice may include text for performing a specific task.

[0195] Electronic device 100 can send user voice to server 1000 (S1020).

[0196] Server 1000 can convert user speech into text (S1025). In this case, server 1000 can use an automatic speech recognition (ASR) module to convert user speech into text.

[0197] Server 1000 can identify (determine) the task corresponding to the user's speech (S1030). Specifically, server 1000 can analyze the user's speech through a natural language understanding (NLU) module to determine the task and entity corresponding to the user's speech.

[0198] Server 1000 can identify (determine) whether the task corresponding to the user's voice is a task that requires user confirmation (S1035). That is, server 1000 can determine whether the task corresponding to the user's voice is related to user privacy or requires security.

[0199] When it is determined that the task corresponding to the user's voice requires user confirmation, the server 1000 may send an inquiry generation command to the electronic device 100 (S1040). In this case, the server 1000 may send information about the user's voice along with the inquiry generation command.

[0200] The electronic device 100 may obtain (generate) an inquiry for user confirmation in response to an inquiry generation command (S1045). In this case, the inquiry may be an inquiry unrelated to the task corresponding to the user's voice.

[0201] The electronic device 100 can receive user responses to user inquiries (S1050) and perform tasks based on the user responses (S1055).

[0202] Figure 11 This is a sequence diagram illustrating another example of a query for user confirmation generated by an artificial intelligence system according to embodiments of the present disclosure. Figure 11 The diagram illustrates an example of how an external server determines whether a task corresponding to a user's voice requires user confirmation and generates a query.

[0203] Reference Figure 11 , Figure 11 S1105, S1110, S1115, S1120, S1125, S1130 and S1135 correspond to in Figure 10 The details of S1005, S1010, S1015, S1020, S1025, S1030 and S1035 described herein are therefore omitted.

[0204] When the task corresponding to the user's voice is identified (determined) as requiring user confirmation, the server 1000 may obtain (generate) a query (S1140). In this case, the server 1000 may generate a query unrelated to the task corresponding to the user's voice.

[0205] Server 1000 can send queries to electronic device 100 (S1145).

[0206] Electronic device 100 can provide an inquiry (S1150). In this case, electronic device 100 can output the inquiry in audio form through speaker 160, but this is only an example, and electronic device 100 can output the inquiry through display 150.

[0207] The electronic device 100 can receive a user response (S1155) and send the received user response to the server 1000 (S1160).

[0208] Server 1000 can identify (determine) whether a user response received from electronic device 100 is a user response corresponding to an inquiry (S1165). When the user response received from electronic device 100 is a user response corresponding to an inquiry, server 1000 can generate a task execution command (a guiding message indicating the result of executing a task together with the task execution command). When the user response received from electronic device 100 is not a user response corresponding to an inquiry, server 1000 can generate an impossible message, wherein the impossible message includes a message indicating that the task cannot be executed.

[0209] Server 1000 may send task execution commands or impossible messages to electronic device 100 (S1170).

[0210] The electronic device 100 can perform a task based on a task execution command or provide an impossible message (S1175).

[0211] Figure 12 This is a sequence diagram illustrating another example of a query for user confirmation generated by an artificial intelligence system according to embodiments of the present disclosure. Figure 12 The image shows an example illustrating how an external server converts a user's speech into text.

[0212] Reference Figure 12 , Figure 12 S1205, S1210, S1215, S1220 and S1225 correspond to in Figure 10 The details of S1005, S1010, S1015, S1020 and S1025 described herein are therefore omitted.

[0213] The server 1000 can send the obtained text to the electronic device 100 (S1230).

[0214] The electronic device 100 can identify (determine) the task corresponding to the user's voice (S1235). Specifically, the electronic device 100 can analyze the user's voice through a natural language understanding (NLU) module to determine the task and entity corresponding to the user's voice.

[0215] The electronic device 100 can identify (determine) whether the task corresponding to the user's voice is a task that requires user confirmation (S1240). That is, the electronic device 100 can determine whether the task corresponding to the user's voice is related to user privacy or requires security.

[0216] When determining whether a task corresponding to a user's voice requires user confirmation, the electronic device 100 may obtain (generate) an inquiry for user confirmation in response to an inquiry generation command (S1245). In this case, the inquiry may be unrelated to the task corresponding to the user's voice.

[0217] The electronic device 100 can receive user responses to user inquiries (S1250) and perform tasks based on the user responses (S1255).

[0218] Figure 13 This is a sequence diagram illustrating an example of an electronic device or external server converting user speech into text based on a security score of the user's speech according to embodiments of the present disclosure.

[0219] Reference Figure 13 , Figure 13 S1305, S1310, S1315, S1320, S1325 and S1330 correspond to in Figure 12 S1205, S1210, S1215, S1220, S1225 and S1230 shown are therefore omitted in detail.

[0220] The electronic device 100 can calculate the security score of the user's voice using a first method based on the obtained text (S1335). In this case, the first method is a linear regression method or a lightweight method based on general rules. As an example, the security score of the user's voice can be calculated as shown in Equation 1.

[0221] Electronic device 100 can determine whether a security score has not been calculated or is within a threshold range (S1340). That is, in cases where it may be impossible to determine the task or entity corresponding to the user's speech by the natural language understanding module present in electronic device 100, or if the security score is within a threshold range (e.g., 0.45 to 0.55) based on a threshold range (e.g., 0.5), electronic device 100 can determine whether a security score has not been calculated or is within a threshold range in order to obtain an accurate security score through external server 1000.

[0222] If the calculated security score is outside the threshold range, the electronic device 100 can obtain (generate) an inquiry based on the calculated security score (S1360). The method for generating an inquiry using the security score has already been described above, so its detailed description is omitted.

[0223] If the security score is not calculated or is within the threshold range, the electronic device 100 may request the server 1000 to recalculate the security score (S1345). In this case, the electronic device 100 may send text information corresponding to the user's voice along with the request.

[0224] Server 1000 may recalculate the security score of the user's voice using a second method (S1350). In this case, the second method may be to use an artificial intelligence model (e.g., a DNN model), wherein the artificial intelligence model is learned to calculate the security score of the user's voice by inputting text corresponding to the user's voice. Server 1000 may send the recalculated security score to electronic device 100 (S1355), and electronic device 100 may obtain (generate) an inquiry based on the recalculated security score (S1360).

[0225] The electronic device 100 can receive user responses to user inquiries (S1365) and perform tasks based on the user responses (S1370).

[0226] The electronic device 100 according to the various embodiments described above can, in conjunction with an external server, determine whether a task corresponding to a user's voice is a task that requires user confirmation, and generate a query for user confirmation based on the determination result.

[0227] exist Figure 13The example described above illustrates how server 1000 converts user speech to text, but this is only an example, and user speech can also be converted to text via ASR module 410 located in electronic device 100.

[0228] The above embodiments have described a case where user voice related to a task requiring user confirmation is input once, but this is only an example, and user voice related to a task requiring user confirmation can be input continuously or periodically.

[0229] In this situation, the electronic device 100 can determine whether the user's voice was input by the same user. Specifically, when the AI ​​agent is activated by triggering voice input, and a second user's voice is input again within a predetermined time after a session for user voice recognition has begun by inputting the first user's voice, the electronic device 100 can determine whether the first user's voice and the second user's voice are from the same user. When the first user's voice and the second user's voice are input by the same user, the electronic device 100 can perform processing on the second user's voice and maintain the session for user voice recognition for a predetermined time. However, if the first user's voice and the second user's voice are not from the same speaker, the electronic device 100 may not process the second user's voice or may provide a rejection response. When the same user does not input any user voice within the predetermined time, the electronic device 100 may terminate the session for user voice recognition.

[0230] Additionally, the electronic device 100 can compare the security score for the first user's voice with the security score for the second user's voice. If the security score for the second user's voice is higher than the security score for the first user's voice, the electronic device 100 can generate an inquiry for additional confirmation (or additional authentication) of the second user's voice. As an example, the electronic device 100 can generate a utterance of leading sentences or words, wherein the sentences or words include text with voice features that distinguish it from the voice features of another user. Furthermore, the electronic device 100 can use data temporarily accumulated in real time while maintaining a session for user voice recognition to perform user authentication.

[0231] The terms “device” or “module” as used in this disclosure can include units configured by hardware, software, or firmware, and can be used compatiblely with terms such as, for example, logic, logic block, component, circuit, etc. The term “device” or “module” can be a component or minimum unit or part thereof that is configured to perform one or more functions. For example, a module can be configured by an application-specific integrated circuit (ASIC).

[0232] Various embodiments of this disclosure may be implemented by software including instructions stored in a machine-readable storage medium (e.g., a computer-readable storage medium). The machine may be an apparatus that invokes stored instructions from the storage medium and operates according to the invoked instructions, and may include an electronic device (e.g., electronic device 100) according to the disclosed embodiments. When an instruction is executed by a processor, the processor may directly perform the function corresponding to the instruction, or other components may perform the function corresponding to the instruction under the control of the processor. Instructions may include code created or executed by a compiler or interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, the term "non-transitory" means that the storage medium is tangible and does not include signals, and does not distinguish whether data is semi-permanent or temporarily stored in the storage medium.

[0233] According to embodiments, methods according to various embodiments disclosed in the documentation may be included and provided in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., an optical disc read-only memory (CD-ROM)) or through an app store (e.g., the Play Store). TM Online distribution. In the case of online distribution, at least a portion of the computer program product may be stored at least temporarily in a storage medium, such as the memory of the manufacturer's server, the app store's server, or a relay server, or may be temporarily created.

[0234] Each of the components (e.g., modules or programs) according to different embodiments may include a single entity or multiple entities, and some of the corresponding sub-components described above may be omitted, or other sub-components may be further included in different embodiments. Optionally or additionally, some components (e.g., modules or programs) may be integrated into one entity and may perform the functions performed by the respective corresponding components prior to integration in the same or similar manner. Operations performed by modules, programs or other components according to different embodiments may be performed sequentially, in parallel, iteratively or heuristically, at least some operations may be performed in a different order or omitted, operations may be combined, or other operations may be added.

[0235] While this disclosure has been shown and described with reference to various embodiments thereof, those skilled in the art will understand that various changes in form and detail may be made therein without departing from the spirit and scope of this disclosure as defined by the appended claims and their equivalents.

Claims

1. An electronic device comprising: Input interface; Communication interface; The memory includes at least one command; as well as At least one processor is connected to an input interface, a communication interface, and a memory, and is configured to control the electronic device. Wherein, the at least one processor executes the at least one command to be configured as follows: Receive user voice input via the input interface. The user's voice is analyzed to identify whether it is related to a task requiring user confirmation. Based on determining that the user's voice is related to a task requiring user confirmation, an inquiry for user confirmation is obtained, and Based on the user response corresponding to the query input through the input interface, a task corresponding to the user's voice is executed. Wherein, the at least one processor executes the at least one command to be further configured as follows: By analyzing the user's voice, tasks related to the user's voice and entities for performing those tasks can be identified. Obtain the security score of the identified task and the security score of the entity. The security score of the user's voice is obtained based on the security score of the identified task and the security score of the entity. The queries are obtained based on the security score of the user's voice, wherein as the security score of the user's voice increases, queries with low relevance to the task corresponding to the user's voice are generated, and as the security score of the user's voice decreases, queries with high relevance to the task corresponding to the user's voice are generated. Among them, queries that are low in relevance to the task corresponding to the user's speech include queries that induce utterances that are semantically far removed from the task.

2. The electronic device according to claim 1, wherein, The inquiry is unrelated to the task corresponding to the user's voice.

3. The electronic device according to claim 1, wherein, The at least one processor executes the at least one command to be further configured as follows: The security score of the user's voice is used to determine whether the user's voice is related to a task that requires user confirmation.

4. The electronic device according to claim 3, wherein, The at least one processor executes the at least one command to be further configured as follows: Based on the security score of the user's voice, it is determined that the user's voice is related to a task that requires user confirmation, and the query is obtained based on the security score of the user's voice.

5. The electronic device according to claim 4, wherein, The at least one processor executes the at least one command to be further configured as follows: Based on the user's voice security score, which is a threshold or higher, an authentication message is provided for user authentication.

6. The electronic device according to claim 1, wherein, The at least one processor executes the at least one command to be further configured as follows: Extract at least one text from the text included in the user's speech, and Obtain queries for inducing user utterances in response to the extracted at least one text.

7. The electronic device according to claim 1, wherein, The at least one processor executes the at least one command to be further configured as follows: By analyzing the user's speech, speech feature information about the text that distinguishes it from the speech feature information of another user is obtained, and The speech feature information about the text is stored in the memory.

8. The electronic device according to claim 7, wherein, The at least one processor executes the at least one command to be further configured as follows: Based on determining that the user's voice is related to a task requiring user confirmation, a query including the stored text is obtained, and Based on the speech feature information about the text obtained from the user response, a task corresponding to the user's speech is performed.

9. The electronic device according to claim 1, wherein, Tasks requiring user confirmation include at least one of the following: remittance tasks, product purchase tasks, email sending tasks, message sending tasks, or telephone call tasks.

10. A method for controlling an electronic device, comprising: Receive user voice messages; The user's voice is analyzed to identify whether it is related to a task that requires user confirmation. Based on determining that the user's voice is related to a task that requires user confirmation, a query for user confirmation is obtained; as well as Based on the user's response corresponding to the query, perform the task corresponding to the user's voice. The step of identifying whether the user's voice is related to a task requiring user confirmation includes: By analyzing the user's voice, tasks related to the user's voice and entities for performing those tasks can be identified. Obtain the security score of the identified task and the security score of the entity. The security score of the user's voice is obtained based on the security score of the identified task and the security score of the entity. The steps for obtaining user confirmation include: The queries are obtained based on the security score of the user's voice, wherein as the security score of the user's voice increases, queries with low relevance to the task corresponding to the user's voice are generated, and as the security score of the user's voice decreases, queries with high relevance to the task corresponding to the user's voice are generated. Among them, queries that are low in relevance to the task corresponding to the user's speech include queries that induce utterances that are semantically far removed from the task.

11. The control method according to claim 10, wherein, The inquiry is unrelated to the task corresponding to the user's voice.

12. The control method according to claim 10, wherein, The steps for identifying whether the user's voice is related to a task requiring user confirmation include: The security score of the user's voice is used to determine whether the user's voice is related to a task that requires user confirmation.

13. The control method according to claim 12, in, The security score of the user's voice is based on determining that the user's voice is related to a task that requires user confirmation.

Images