Electronic apparatus and control method thereof

The electronic device learns from user responses to voice commands, correcting misinterpretations by storing user feedback and providing guide information, thus enhancing its ability to accurately respond to voice commands.

KR103044908B1Active Publication Date: 2026-11-06SAMSUNG ELECTRONICS CO LTD
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
KR1020200022900
Authority / Receiving Office
KR · KR
Patent Type
Patents
Current Assignee / Owner
Filing Date
2020-02-25
Publication Date
2026-11-06
Estimated Expiration
2040-02-25

AI Technical Summary

Technical Problem

Electronic devices equipped with Voice Assistant Platforms often fail to perfectly recognize user voice commands, leading to incorrect actions or no actions at all, making it difficult for users to issue new commands and identify malfunctioning devices.

Method used

The device includes a processor that stores user response information when a voice command is followed by a user input within a threshold time, and provides guide information if subsequent voice commands match the initial command or express negative emotions, allowing the device to learn and correct misinterpretations.

Benefits of technology

Enhances user convenience by improving the device's ability to recognize and respond correctly to voice commands, providing guide information to correct misinterpretations and learn from user responses.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 112020019784072-PAT00001_ABST
    Figure 112020019784072-PAT00001_ABST
Patent Text Reader

Abstract

An electronic device is disclosed. The electronic device includes a microphone, an input unit, a memory, a display, and a processor connected to the microphone, the input unit, the memory, and the display to control the electronic device. When a first user voice is received through the microphone, the processor performs an operation corresponding to the first user voice. When a user command is received through the input unit within a first threshold time from the time when the first user voice is received or the time when the operation corresponding to the first user voice is performed, the processor stores user response information including information about the operation corresponding to the first user voice and the user command in the memory. When a second user voice is received through the microphone, the processor performs an operation corresponding to the second user voice. If the type of the operation corresponding to the first user voice and the type of the operation corresponding to the second user voice are the same, the processor can control the display to provide guide information corresponding to the user command based on the user response information stored in the memory.
Need to check novelty before this filing date? Find Prior Art

Description

Technology Field

[0001] The present disclosure relates to an electronic device and a method for controlling the same, and more specifically, to an electronic device and a method for controlling the same that perform an operation according to a user's voice command. Background Technology

[0002] With the recent advancement of technology, the number of electronic devices equipped with Voice Assistant Platforms is increasing. Electronic devices can provide various services through the Voice Assistant Platform. In particular, electronic devices can recommend utterances related to the user's speech. For example, when an electronic device receives a user's speech, it can provide a list of recommended utterances related to the previous speech. Additionally, the electronic device may provide the list of recommended utterances by further considering at least one of the current system status, whether an app is running, location, or changes in network connection status.

[0003] However, the electronic device cannot perfectly recognize the user's voice command 100%. In this case, the electronic device performs an action that does not correspond to the user's voice command or does not perform any action.

[0004] Since the user does not have information on which utterances the electronic device supports, there is a problem in that it is difficult to utter new voice commands if the electronic device performs an action that does not correspond to the user's voice command or performs no action at all.

[0005] In addition, there is also a problem where the electronic device fails to recognize a situation where an issue has occurred because it performs actions that do not correspond to the user's voice command or performs no actions at all.

[0006] Accordingly, there is a need to develop methods to identify when an electronic device is malfunctioning and to improve it. The problem to be solved

[0007] The present disclosure is in accordance with the aforementioned necessity, and the purpose of the present disclosure is to provide an electronic device and a control method thereof for identifying whether an operation in response to a user's voice command is a malfunction and improving such malfunction. means of solving the problem

[0008] According to one embodiment of the present disclosure for achieving the above objectives, the electronic device comprises a microphone, an input unit, a memory, a display, and a processor connected to the microphone, the input unit, the memory, and the display to control the electronic device. The processor performs an operation corresponding to the first user voice when a first user voice is received through the microphone, and if a user command is received through the input unit within a first threshold time from the time when the first user voice is received or the time when the operation corresponding to the first user voice is performed, the processor stores user response information including information on the operation corresponding to the first user voice and the user command in the memory, and if a second user voice is received through the microphone, the processor performs an operation corresponding to the second user voice, and if the type of the operation corresponding to the first user voice and the type of the operation corresponding to the second user voice are the same, the processor controls the display to provide guide information corresponding to the user command based on the user response information stored in the memory.

[0009] Additionally, the processor may identify the type of the second user voice if the second user voice is received through the microphone within a second threshold time from the time when the operation corresponding to the first user voice is performed, and if the type of the second user voice is a preset type and the user command is received, store the user response information including information about the operation corresponding to the first user voice and the user command in the memory.

[0010] In addition, when the processor receives the second user voice of the preset type and then receives the user command, it can store the user response information, including information on the operation corresponding to the first user voice and the user command, in the memory.

[0011] In addition, the second user voice of the above-mentioned preset type may be a user voice containing words expressing negative emotions.

[0012] In addition, the processor can control the display to display content corresponding to the second user voice and to display the guide information overlaid on the content.

[0013] And, the processor can stop the playback of the content while the guide information is being displayed.

[0014] Additionally, the processor can perform an operation corresponding to the user command when the second user voice is received while the number of times the user response information is stored in the memory is greater than or equal to a threshold number.

[0015] In addition, the guide information may include at least one of a message that induces an action corresponding to the user command or an item for executing an action corresponding to the user command.

[0016] Additionally, if the processor does not receive a user command through the input unit within a third threshold time after the guide information is provided, it may delete the user response information corresponding to the first user voice from the memory.

[0017] In addition, the processor further includes a communication interface, and when the user command is received, the processor controls the communication interface to transmit information regarding an action corresponding to the first user voice and the user command to a server, receives the user response information from the server through the communication interface, and can store the received user response information in the memory.

[0018] In addition, information regarding the operation corresponding to the first user voice may include information regarding the type of operation that the electronic device will perform based on the first user voice.

[0019] Meanwhile, according to one embodiment of the present disclosure, a control method for an electronic device comprises: a step of performing an operation corresponding to the first user voice when a first user voice is received; a step of storing user response information including information on the operation corresponding to the first user voice and the user command when a user command is received within a first threshold time from the time when the first user voice is received or the time when the operation corresponding to the first user voice is performed; a step of performing an operation corresponding to the second user voice when a second user voice is received; and a step of providing guide information corresponding to the user command based on the user response information when the type of the operation corresponding to the first user voice and the type of the operation corresponding to the second user voice are the same.

[0020] Additionally, the method further includes a step of identifying the type of the second user voice when the second user voice is received within a second threshold time from the time when the action corresponding to the first user voice is performed, and the storing step may store the user response information including information about the action corresponding to the first user voice and the user command when the type of the second user voice is a preset type and the user command is received.

[0021] In addition, the above-mentioned saving step may store user response information including information on an action corresponding to the first user voice and the user command when the user command is received after the second user voice of the above-mentioned preset type is received.

[0022] In addition, the second user voice of the above-mentioned preset type may be a user voice containing words expressing negative emotions.

[0023] And, the step of performing the action corresponding to the second user voice may display content corresponding to the second user voice, and the step of providing may display the guide information by overlaying it on the content.

[0024] In addition, the step provided above can stop the playback of the content while the guide information is displayed.

[0025] In addition, the step provided above can perform an operation corresponding to the user command when the second user voice is received while the number of times the user response information is stored is greater than or equal to a threshold number.

[0026] Additionally, the guide information may include at least one of a message that induces an action corresponding to the user command or an item for executing an action corresponding to the user command.

[0027] And, if a user command is not received within a third threshold time after the above guide information is provided, the method may further include a step of deleting the user response information corresponding to the first user voice.

[0028] Additionally, the above-mentioned storing step may include the step of transmitting information regarding an action corresponding to the first user voice and the user command to a server when the user command is received, the step of receiving the user response information from the server, and the step of storing the received user response information. Effects of the invention

[0029] According to various embodiments of the present disclosure as described above, an electronic device can improve user convenience by learning the user's response following a user's voice command and providing guide information based on the learned user response when the same voice command is input thereafter. Brief explanation of the drawing

[0030] FIG. 1 is a block diagram showing the configuration of an electronic device according to one embodiment of the present disclosure. FIGS. 2a and FIGS. 2b are drawings for illustrating a learning operation according to one embodiment of the present disclosure. FIGS. 3a and FIGS. 3b are drawings for illustrating a learning operation according to another embodiment of the present disclosure. FIGS. 4a and 4b are drawings for illustrating a learning operation for various user voices according to one embodiment of the present disclosure. FIG. 5 is a drawing for explaining an operation based on user response information according to one embodiment of the present disclosure. FIG. 6 is a drawing for explaining a user setting operation according to one embodiment of the present disclosure. FIG. 7 is a drawing for explaining the advantages of using a server according to one embodiment of the present disclosure. FIG. 8 is a flowchart illustrating a method for controlling an electronic device according to one embodiment of the present disclosure. Specific details for implementing the invention

[0031] The present disclosure will be described in detail below with reference to the attached drawings.

[0032] The terms used in the embodiments of this disclosure have been selected to be as widely used as possible, taking into account their functions within this disclosure; however, these terms may vary depending on the intent of those skilled in the art, case law, the emergence of new technologies, etc. Additionally, in specific cases, terms have been arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the relevant explanatory section of this disclosure. Therefore, terms used in this disclosure should be defined not merely by their names, but based on their meanings and the overall content of this disclosure.

[0033] In this specification, expressions such as “have,” “may have,” “include,” or “may include” indicate the presence of such features (e.g., numerical values, functions, operations, or components such as parts) and do not exclude the presence of additional features.

[0034] The expression "at least one of A or / and B" should be understood as representing either "A" or "B" or "A and B".

[0035] Expressions such as "first," "second," "first," or "second" used in this specification may modify various components regardless of order and / or importance, and are used only to distinguish one component from another and do not limit said components.

[0036] The singular expression includes the plural expression unless the context clearly indicates otherwise. In this application, terms such as "comprising" or "consisting of" are intended to specify the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0037] In this specification, the term "user" may refer to a person using an electronic device or a device using an electronic device (e.g., an artificial intelligence electronic device).

[0038] An embodiment of the present disclosure will be described in more detail below with reference to the attached drawings.

[0039] FIG. 1 is a block diagram showing the configuration of an electronic device according to one embodiment of the present disclosure.

[0040] The electronic device (100) is a device that performs operations according to user voice and may be a device that is equipped with a display such as a TV, desktop PC, laptop, video wall, LFD (large format display), Digital Signage, DID (Digital Information Display), projector display, DVD (digital video disk) player, smartphone, tablet PC, monitor, smart glasses, smart watch, etc., and directly displays the acquired graphic image.

[0041] Alternatively, the electronic device (100) may be a device that is not equipped with a display, such as a set-top box (STB), a speaker, or a computer main body, and may be a device that provides a graphic image representing an action according to the user's voice to a display device. However, it is not limited thereto, and the electronic device (100) may be any device that performs an action according to the user's voice.

[0042] According to FIG. 1, the electronic device (100) includes a microphone (110), an input unit (120), a memory (130), a display (140), and a processor (150). However, it is not limited thereto, and the electronic device (100) may be implemented with some components excluded.

[0043] A microphone (110) is configured to receive sound input and convert it into an audio signal. The microphone (110) is electrically connected to a processor (150) and can receive sound under the control of the processor (150). Here, the sound may include user voice and ambient noise of the electronic device (100).

[0044] The microphone (110) may be implemented in a form integrated with the upper side, front direction, side direction, etc. of the electronic device (100). Alternatively, the microphone (110) may be provided with a remote control, etc., separate from the electronic device (100). In this case, the remote control may receive sound through the microphone (110) and provide the received sound to the electronic device (100).

[0045] The microphone (110) may include various configurations such as a microphone that collects analog sound, an amplifier circuit that amplifies the collected sound, an A / D conversion circuit that samples the amplified sound and converts it into a digital signal, and a filter circuit that removes noise components from the converted digital signal.

[0046] The microphone (110) may include multiple sub-microphones. For example, the microphone (110) may include one sub-microphone each on the front, back, left, and right sides of the electronic device (100). However, it is not limited thereto, and the electronic device (100) may include only one microphone (110).

[0047] The microphone (110) may also be implemented in the form of a sound sensor.

[0048] The input unit (120) is configured to receive user commands. For example, the input unit (120) may be implemented as a communication interface to receive control signals from a remote control device. In this case, the user can control the electronic device (100) by inputting user commands through the remote control device.

[0049] Here, the communication interface is a configuration that performs communication with various types of external devices according to various types of communication methods. The communication interface may include at least one of a Wi-Fi module, a Bluetooth module, an infrared communication module, or a wireless communication module. Here, each communication module may be implemented in the form of at least one hardware chip.

[0050] The Wi-Fi module and Bluetooth module perform communication using the Wi-Fi and Bluetooth methods, respectively. When using the Wi-Fi or Bluetooth module, various connection information, such as the SSID and session key, is transmitted and received first; after establishing a communication connection using this information, various data can be transmitted and received. The infrared communication module performs communication according to infrared communication (IrDA, Infrared Data Association) technology, which wirelessly transmits data over short distances using infrared rays that lie between visible light and millimeter waves.

[0051] In addition to the communication method described above, the wireless communication module may include at least one communication chip that performs communication according to various wireless communication standards such as Zigbee, 3G (3rd Generation), 3GPP (3rd Generation Partnership Project), LTE (Long Term Evolution), LTE-A (LTE Advanced), 4G (4th Generation), and 5G (5th Generation).

[0052] In addition, the communication interface may include at least one of a LAN (Local Area Network) module, an Ethernet module, or a wired communication module that performs communication using a pair cable, a coaxial cable, or a fiber optic cable.

[0053] As another example, the input unit (120) may be implemented in a form that directly receives user commands as various types of buttons, such as mechanical buttons, touch pads, and wheels, formed in any area of ​​the exterior of the main body of the electronic device (100), such as the front, side, or back portions.

[0054] Alternatively, the input unit (120) may be implemented as a camera to receive user interactions such as user motion. Alternatively, the input unit (120) may be implemented in a form including a microphone (110). In this case, the user may input user commands into the electronic device (100) by speaking user voice.

[0055] However, the above examples are merely examples, and the input unit (120) may be of any form as long as it is capable of receiving user commands.

[0056] The memory (130) can store user response information. The user response information may include information about the user's response within a threshold time after an action based on the user's voice, and a detailed explanation thereof will be provided later.

[0057] The memory (130) may be implemented as non-volatile memory and volatile memory, but is not limited thereto. For example, a hard disk may be used instead of the memory (130), and any configuration capable of storing data is possible.

[0058] The display (140) is a configuration for displaying graphic data. For example, the display (140) is a configuration for displaying guide information corresponding to user commands and can be implemented as various types of displays such as LCD (Liquid Crystal Display), OLED (Organic Light Emitting Diodes) display, PDP (Plasma Display Panel), micro LED, Laser Display, VR, Glass, etc. The display (140) may also include a driving circuit, a backlight unit, etc., which can be implemented in forms such as a-si TFT, LTPS (low temperature poly silicon) TFT, OTFT (organic TFT), etc. Meanwhile, the display (140) can be implemented as a touch screen combined with a touch sensor, a flexible display, a 3D display, etc.

[0059] The processor (150) controls the overall operation of the electronic device (100). Specifically, the processor (150) can control the overall operation of the electronic device (100) by being connected to each component of the electronic device (100). For example, the processor (150) can control the operation of the electronic device (100) by being connected to components such as a microphone (110), an input unit (120), a memory (130), a display (140), etc.

[0060] According to one embodiment, the processor (150) may be implemented as a digital signal processor (DSP), a microprocessor, or a time controller (TCON). However, it is not limited thereto, and may include or be defined by one or more of a central processing unit (CPU), a micro controller unit (MCU), a micro processing unit (MPU), a controller, an application processor (AP), a communication processor (CP), or an ARM processor. Additionally, the processor (150) may be implemented as a System on Chip (SoC) or a large-scale integration (LSI) with a built-in processing algorithm, or may be implemented in the form of a Field Programmable Gate Array (FPGA).

[0061] The processor (150) can perform an action corresponding to the first user voice when the first user voice is received through the microphone (110). For example, when the processor (150) receives the user voice saying "Play channel 2," it can display channel 2 to correspond to the user voice.

[0062] However, the processor (150) may fail to recognize voice or may perform voice recognition incorrectly. For example, if the processor (150) receives a user voice saying "Turn on channel 2" but does not recognize it, it may not perform any action or may provide a guidance message such as "Please say it again." Alternatively, even if the processor (150) receives a user voice saying "Turn on channel 2," it may identify it as "Turn on channel 1" and display channel 1. For convenience of explanation, the following description assumes that the processor (150) identifies the user voice saying "Turn on channel 2" as "Turn on channel 1."

[0063] In this case, since the desired channel is not displayed, the user may provide a user command to the electronic device (100) to display channel 2 in a different way. For example, the user may provide a user command to the electronic device (100) to display channel 2 by pressing the number 2 key of the remote control device. Alternatively, the user may provide a user command to the electronic device (100) by additionally speaking a user voice.

[0064] The processor (150) may store user response information, including information about an action corresponding to the first user voice and a user command, in the memory (130) if a user command is received through the input unit (120) within a first threshold time from the time when the first user voice is received or the time when an action corresponding to the first user voice is performed. Here, the information about the action corresponding to the first user voice may include information about the type of action to be performed by the electronic device (100) based on the first user voice. For example, the processor (150) may perform an action of changing a channel, an action of searching for content, an action of searching for a word in a dictionary, etc., according to the first user voice, and in this case, the information about the action corresponding to the first user voice may include at least one of an action of changing a channel, an action of searching for content, or an action of searching for a word in a dictionary.

[0065] More specifically, for example, if the processor (150) receives a user command to display channel 2 through the input unit (120) within 5 seconds from the time when a user voice such as "Turn on channel 2" is received or the channel is changed to channel 1 according to the user voice, the processor (150) can store user response information including information that channel 1 was displayed based on the first user voice and a user command to display channel 2 in the memory (130).

[0066] That is, the processor (150) determines that there is a problem with the operation according to the first user voice only when a user command is received within a first threshold time from the time when the first user voice is received or the time when an operation corresponding to the first user voice is performed, and stores the user response information along with the user command in memory (130).

[0067] Here, the processor (150) can match information about an action corresponding to the first user voice with a user command and store it as user response information. That is, the user response information may include a plurality of information pairs.

[0068] For example, user response information may include information for displaying channel 1 and a user command for displaying channel 2 as a first information pair, and information for displaying channel 11 and a user command for displaying channel 12 as a second information pair, and may include any other various information pairs.

[0069] After user response information is stored in memory (130), the processor (150) performs an action corresponding to the second user voice when a second user voice is received through a microphone, and can identify whether the type of action corresponding to the first user voice and the type of action corresponding to the second user voice are the same. Then, if the type of action corresponding to the first user voice and the type of action corresponding to the second user voice are the same, the processor (150) can control the display (140) to provide guide information corresponding to a user command based on the user response information stored in memory (130).

[0070] In the example described above, when the processor (150) receives a user voice such as "Play channel 2," it may incorrectly identify it and display channel 1. Then, the processor (150) can identify whether the action of displaying channel 1 corresponds to the first user voice stored in the user response information. That is, if the processor (150) identifies that the action of displaying channel 1 is information stored in the user response information, it can control the display (140) to provide guide information corresponding to the user command to display channel 2.

[0071] Here, the guide information may include at least one of a message that induces an action corresponding to a user command or an item for executing an action corresponding to a user command. In the example described above, the guide information may include at least one of a message such as "Display channel 2" or an item indicating "channel 2". The user may change channel 1 to channel 2 by speaking the user's voice in the same way as the message. Alternatively, the user may change channel 1 to channel 2 by selecting an item through a remote control device, etc.

[0072] Meanwhile, although it has been described above that user response information is stored in memory (130) when a user command is received within a first threshold time from the time when the first user voice is received or the time when an action corresponding to the first user voice is performed, this is merely one embodiment.

[0073] For example, the processor (150) may identify the type of the second user voice if the second user voice is received through the microphone (110) within a second threshold time from the time when the action corresponding to the first user voice is performed, and if the type of the second user voice is a preset type and a user command is received, it may store user response information including information about the action corresponding to the first user voice and the user command in the memory (130).

[0074] That is, the processor (150) may store user response information in memory (130) if a second user voice of a preset type is received within a second threshold time from the time when the first user voice is received.

[0075] In particular, the processor (150) may store user response information, including information on an action corresponding to the first user voice and a user command, in memory (130) when a user command is received after a second user voice of a preset type is received. In this case, the first threshold time is greater than the second threshold time.

[0076] However, it is not limited to this, and user response information may also be stored in memory (130) even when the first threshold time is shorter than the second threshold time. For example, when a second user voice of a preset type is received after a user command is received, the processor (150) may store user response information including information on an action corresponding to the first user voice and a user command in memory (130).

[0077] A second user voice of a preset type may be a user voice containing words expressing negative emotions. For example, the second user voice may be a user voice containing words expressing negative emotions, such as "It's not working well" or "No, not that."

[0078] Meanwhile, the processor (150) can control the display (140) to display content corresponding to the second user voice and to display guide information overlaid on the content.

[0079] In the example described above, the processor (150) can control the display (140) to display content provided by channel 1 and to overlay guide information, such as a message like “Display channel 2” or an item indicating “channel 2,” onto the content. In this case, the processor (150) may stop the playback of the content while the guide information is being displayed.

[0080] Meanwhile, the above assumes that the number of times a user command is received within a first threshold time from the time when the first user voice is received or the time when an action corresponding to the first user voice is performed is one. However, such malfunction may be repeated several times, and the processor (150) may store more of the same user response information in memory (130).

[0081] Additionally, when the processor (150) receives a second user voice while the number of times the same user response information is stored in memory (130) is greater than or equal to a threshold number, it may perform an action corresponding to a user command. In this case, the processor (150) may not perform an action of providing guide information.

[0082] If a user command is not received through the input unit (120) within a third threshold time after the guide information is provided, the processor (150) may delete the user response information corresponding to the first user voice from the memory (130). However, this is not limited thereto, and the user response information corresponding to the first user voice may be deleted only if the number of times a user command is not received within the third threshold time after the guide information is provided is greater than or equal to the threshold number.

[0083] In the above description, it has been explained that the electronic device (100) directly obtains user response information, but it is not limited thereto. For example, the electronic device (100) further includes a communication interface, and the processor (150) controls the communication interface to transmit information about an action corresponding to the first user voice and the user command to a server when a user command is received, and may receive user response information from the server through the communication interface and store the received user response information in memory (130).

[0084] In this case, the server can collect various user response information from multiple other electronic devices as well as the electronic device (100).

[0085] As described above, the electronic device (100) learns the user's response after the user's voice command, and can provide guide information based on the learned user's response when the same voice command is input thereafter.

[0086] The operation of the electronic device (100) will be explained in more detail below through drawings.

[0087] FIGS. 2a and FIGS. 2b are drawings for illustrating a learning operation according to one embodiment of the present disclosure.

[0088] First, the processor (150) can receive a first user voice, such as "Play channel 2," through the microphone (110) as shown in FIG. 2a. However, the processor (150) may incorrectly recognize this and display channel 1.

[0089] As illustrated in FIG. 2b, the processor (150) can first display channel 2 if it receives a control command from a remote control device to display channel 2 within a first threshold time from the time when the first user voice is received or the time when an action corresponding to the first user voice is performed.

[0090] And, the processor (150) can store user response information in memory (130) including information for displaying channel 1 based on the first user voice and a control command for displaying channel 2.

[0091] Here, the processor (150) may not store the first user voice itself as user response information. For example, if a first user voice such as "Turn on channel 2" is matched with a control command to display channel 2 and stored as user response information, the user voice cannot be identified with "Turn on channel 2." For instance, if the processor (150) inputs user voice such as "Channel 2" or "Turn on 2," it may not be able to identify the corresponding user response information. Therefore, the processor (150) matches an action based on the first user voice, rather than the first user voice itself, with a user command and stores it as user response information.

[0092] FIGS. 3a and FIGS. 3b are drawings for illustrating a learning operation according to another embodiment of the present disclosure.

[0093] First, the processor (150) can receive a first user voice, such as "Play channel 2," through the microphone (110) as shown in FIG. 3a. However, the processor (150) may incorrectly recognize this and display channel 1.

[0094] And, the processor (150) receives a second user voice, such as "No. Not that," through the microphone (110) within a second threshold time from the time when an action corresponding to the first user voice is performed, and can identify that the second user voice includes words expressing negative emotions.

[0095] As illustrated in FIG. 3b, the processor (150) can first display channel 2 if it receives a control command from a remote control device to display channel 2 within a first threshold time from the time when the second user voice contains a word expressing negative emotion or the time when the first user voice is received or the time when an action corresponding to the first user voice is performed.

[0096] And, the processor (150) can store user response information in memory (130) including information for displaying channel 1 based on the first user voice and a control command for displaying channel 2.

[0097] That is, the processor (150) may store user response information after performing an action according to the first user voice, if there is a negative response from the user and a user command is received.

[0098] In this case, the reliability of user response information can be improved compared to the cases of Figs. 2a and 2b.

[0099] FIGS. 2a to 3b describe the case where a user utters a user voice changing the channel. FIGS. 4a and 4b describe the learning operation of the processor (150) according to a wider variety of user voices.

[0100] FIGS. 4a and 4b are drawings for illustrating a learning operation for various user voices according to one embodiment of the present disclosure.

[0101] First, the processor (150) can receive a first user voice, such as “Find content A,” through the microphone (110), as shown at the top of FIG. 4a.

[0102] As shown at the bottom of FIG. 4a, the processor (150) may display a notification message such as “Related content not found” because no search result for content A is identified. That is, the processor (150) may not be able to provide the desired result to the user. This may be because the database where the processor (150) performed the search is incorrect. For example, if database A stores information about content A and database B does not store information about content A, and the processor (150) performs a search only in database B, the search result may not be identified.

[0103] And, the processor (150) may receive a second user voice, such as “Run App A,” through the microphone (110) within a first threshold time from the time when the first user voice is received or the time when an action corresponding to the first user voice is performed. In this case, the processor (150) executes App A and may store user response information, including information on the search operation of content and an execution command for App A based on the first user voice, in the memory (130).

[0104] FIG. 4a illustrates a case where searching for content A is impossible, but it is not limited thereto. For example, the processor (150) may perform a search for content A in App A' and display the search results in App A'. Here, App A' is an application that provides an introductory video for content A, and the user may want to run App A, which provides content A itself. After displaying the search results in App A', the processor (150) may receive a second user voice, such as “Run App A.” In this case, the processor (150) runs App A and may store user response information, including information on the search operation of the content and a command to run App A, in memory (130) based on the first user voice.

[0105] Alternatively, the processor (150) may receive a first user voice, such as “MLB.”, through the microphone (110), as shown at the top of FIG. 4b.

[0106] The processor (150) can search for information about MLB in a dictionary and display the search results, as shown at the bottom of FIG. 4b. The user may have intended to watch a sports channel related to MLB, but the processor (150) may not have provided the desired result because it only searches for dictionary meanings.

[0107] The processor (150) may receive a second user voice, such as “Run App B,” through the microphone (110) within a first threshold time from the time when the first user voice is received or the time when an action corresponding to the first user voice is performed. App B may be an application that provides various broadcast channels.

[0108] In this case, the processor (150) executes App B and can store user response information in memory (130), including information about the action of searching for a word in a dictionary based on the first user voice and an execution command for App B.

[0109] As described above, if the action desired by the user is not performed, the processor (150) can identify the user's response and store it in memory (130) as user response information.

[0110] FIG. 5 is a diagram illustrating an operation based on user response information according to an embodiment of the present disclosure. FIG. 5 assumes a state in which user response information described in FIG. 2a to FIG. 3b is stored. That is, the user response information is described as including information for displaying channel 1 based on a first user voice and a control command for displaying channel 2.

[0111] As illustrated in FIG. 5, the processor (150) can perform an action corresponding to the second user voice when a second user voice, such as "Play channel 2," is received through the microphone (110). Here, the action corresponding to the second user voice may also be an action of displaying channel 1 due to misrecognition. If the processor (150) correctly recognizes the second user voice and displays channel 2, no additional action is performed because there is no information in the user response information that channel 2 was displayed.

[0112] If the type of action corresponding to the first user voice and the type of action corresponding to the second user voice are the same, the processor (150) may provide guide information corresponding to the user command based on user response information stored in memory (130). The guide information may include at least one of "channel 2" item (510), a message (520) such as "play channel 2," or "STB channel 2" item (530).

[0113] Meanwhile, although the first user voice in FIGS. 2a to 3b and the second user voice in FIG. 5 were both described as "Turn on channel 2," only the first user voice in FIGS. 2a to 3b may be "Turn on channel 2," and the second user voice in FIG. 5 may be "Turn on 2." Here, if the processor (150) misrecognizes the second user voice, such as "Turn on 2," and displays channel 1, the same guide information as described above may be provided. That is, even if the first user voice and the second user voice are different, if the corresponding operation is the same, the processor (150) may provide the same guide information. This is because the processor (150) stores the operation corresponding to the first user voice, rather than the first user voice itself, as user response information. Alternatively, if the processor (150) does not misrecognize the second user voice, such as "Turn on 2," and displays channel 2, there is no problem and no additional operation is performed.

[0114] Alternatively, only the first user voice in FIGS. 2a to 3b may be "Play channel 2," and the second user voice in FIG. 5 may be "Play channel 12." Here, the processor (150) may misrecognize the second user voice, such as "Play channel 12," and display channel 1. In this case, the processor (150) provides guide information such as the bottom of FIG. 5, but this is unrelated to channel 12 that the user wants. However, this guide information is merely displayed and disappears if no user command is entered for a certain period of time, thereby minimizing inconvenience to the user.

[0115] FIG. 6 is a diagram illustrating a user setting operation according to an embodiment of the present disclosure. The user setting screen of FIG. 6 assumes cases where user response information is generated according to cases where a mute command is received after a first user voice is received, where a movie mode command is received after a first user voice is received, and where a mirroring command is received after a first user voice is received.

[0116] The processor (150) may provide a user setting screen as shown in FIG. 6. For example, an icon for displaying a user setting screen may be provided on one side of the screen in FIG. 5, and the user may select it to receive a screen such as FIG. 6.

[0117] And, when one of the menus provided in the user setting screen of FIG. 6 is selected and a user voice that performs the same operation as the first user voice is received, the processor (150) may perform an operation according to the selected function without providing guide information.

[0118] FIG. 7 is a drawing for explaining the advantages of using a server (200) according to one embodiment of the present disclosure.

[0119] The processor (150) may also manage user response information by communicating with the server (200), as shown in FIG. 7.

[0120] For example, if a user command is received within a first threshold time from the time when the first user voice is received or the time when the action corresponding to the first user voice is performed, the processor (150) transmits information about the action corresponding to the first user voice and the user command to the server (200), receives user response information from the server (200), and can store the received user response information in memory (130). Subsequently, if the second user voice is received, the processor (150) performs an action corresponding to the second user voice, and if the type of the action corresponding to the first user voice and the type of the action corresponding to the second user voice are the same, it can provide guide information corresponding to the user command based on the user response information stored in memory (130).

[0121] Alternatively, the processor (150) may transmit information regarding the action corresponding to the first user voice and the user command to the server (200) if the user command is received within a first threshold time from the time when the first user voice is received or the time when the action corresponding to the first user voice is performed. The server (200) may obtain user response information including information regarding the action corresponding to the first user voice and the user command. Subsequently, when the second user voice is received, the processor (150) may perform an action corresponding to the second user voice and transmit information regarding the action corresponding to the second user voice to the server (200). The server (200) may identify whether the action corresponding to the second user voice is stored as user response information, and if it is stored, identify the user command stored by matching it with the action corresponding to the second user voice, and transmit guide information corresponding to the identified user command to the electronic device (100). The processor (150) may provide the received guide information.

[0122] FIG. 8 is a flowchart illustrating a method for controlling an electronic device according to one embodiment of the present disclosure.

[0123] First, when a first user voice is received, an action corresponding to the first user voice is performed (S810). Then, if a user command is received within a first threshold time from the time when the first user voice is received or the time when the action corresponding to the first user voice is performed, user response information including information about the action corresponding to the first user voice and the user command is stored (S820). Then, when a second user voice is received, an action corresponding to the second user voice is performed (S830). Then, if the type of the action corresponding to the first user voice and the type of the action corresponding to the second user voice are the same, guide information corresponding to the user command is provided based on the user response information (S840).

[0124] Here, if a second user voice is received within a second threshold time from the time when an action corresponding to the first user voice is performed, the method further includes a step of identifying the type of the second user voice, and the storage step (S820) can store user response information including information about an action corresponding to the first user voice and a user command if the type of the second user voice is a preset type and a user command is received.

[0125] And, the saving step (S820) can store user response information including information about an action corresponding to the first user voice and a user command when a user command is received after a second user voice of a preset type is received.

[0126] Here, the second user voice of the preset type may be a user voice containing words expressing negative emotions.

[0127] Meanwhile, the step of performing an action corresponding to the second user voice (S830) displays content corresponding to the second user voice, and the step of providing (S840) can display guide information by overlaying it on the content.

[0128] Here, the providing step (S840) can stop the playback of the content while the guide information is displayed.

[0129] Meanwhile, the providing step (S840) can perform an action corresponding to a user command when a second user voice is received while the number of times user response information is stored is greater than or equal to a threshold number.

[0130] In addition, the guide information may include at least one of a message that induces an action corresponding to a user command or an item for executing an action corresponding to a user command.

[0131] Meanwhile, if a user command is not received within a third threshold time after the guide information is provided, the method may further include a step of deleting user response information corresponding to the first user voice.

[0132] And, the saving step (S820) may include the step of transmitting information about an action corresponding to the first user voice and the user command to a server when a user command is received, the step of receiving user response information from the server, and the step of saving the received user response information.

[0133] According to various embodiments of the present disclosure as described above, an electronic device can improve user convenience by learning the user's response following a user's voice command and providing guide information based on the learned user response when the same voice command is input thereafter.

[0134] Meanwhile, according to the exemplary embodiments of the present disclosure, the various embodiments described above may be implemented as software comprising instructions stored on a machine-readable storage medium (e.g., a computer). The machine may include an electronic device (e.g., electronic device (A)) according to the disclosed embodiments, which is a device capable of calling instructions stored from the storage medium and operating according to the called instructions. When instructions are executed by a processor, the processor may perform a function corresponding to the instructions directly or by using other components under the control of the processor. Instructions may include code generated or executed by a compiler or an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, "non-transitory" means only that the storage medium does not contain a signal and is tangible, and does not distinguish whether data is stored semi-permanently or temporarily on the storage medium.

[0135] Additionally, according to one embodiment of the present disclosure, the method according to the various embodiments described above may be provided as included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or online through an application store (e.g., Play Store™). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created in a storage medium such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0136] Additionally, according to one embodiment of the present disclosure, the various embodiments described above may be implemented in a recording medium readable by a computer or a similar device using software, hardware, or a combination thereof. In some cases, the embodiments described herein may be implemented as the processor itself. According to a software implementation, embodiments such as the procedures and functions described herein may be implemented as separate software modules. Each of the software modules may perform one or more functions and operations described herein.

[0137] Meanwhile, computer instructions for performing processing operations of the device according to the various embodiments described above may be stored in a non-transitory computer-readable medium. When computer instructions stored in such a non-transitory computer-readable medium are executed by the processor of a specific device, they cause the specific device to perform processing operations in the device according to the various embodiments described above. A non-transitory computer-readable medium refers to a medium that stores data semi-permanently and is readable by a device, rather than a medium that stores data for a short period of time, such as a register, cache, or memory. Specific examples of a non-transitory computer-readable medium may include CDs, DVDs, hard disks, Blu-ray discs, USBs, memory cards, ROMs, etc.

[0138] Additionally, each component (e.g., module or program) according to the various embodiments described above may be composed of a single or multiple entities, and some of the aforementioned sub-components may be omitted, or other sub-components may be further included in the various embodiments. Generally or additionally, some components (e.g., module or program) may be integrated into a single entity to perform the same or similar functions as those performed by each of the respective components prior to integration. The operations performed by the module, program, or other components according to the various embodiments may be executed sequentially, in parallel, iteratively, or heuristically, or at least some operations may be executed in a different order, omitted, or other operations added.

[0139] Although preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above. It is understood that various modifications can be made by those skilled in the art without departing from the essence of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical spirit or perspective of the present disclosure. Explanation of the symbols

[0140] 100 : Electronic device 110 : Microphone 120 : Input section 130 : Memory 140 : Display 150 : Processor

Claims

Claim 1 An electronic device comprises: a microphone; an input unit; a memory; a display; and a processor connected to the microphone, the input unit, the memory, and the display to control the electronic device; wherein the processor performs an operation corresponding to the first user voice when a first user voice is received through the microphone, and if a user command is received through the input unit within a first threshold time from the time when the first user voice is received or the time when the operation corresponding to the first user voice is performed, the processor stores user response information including information on the operation corresponding to the first user voice and the user command in the memory, and performs an operation corresponding to the second user voice when a second user voice is received through the microphone, and if the type of the operation corresponding to the first user voice and the type of the operation corresponding to the second user voice are the same, the processor controls the display to provide guide information that induces an operation according to the user command based on the user response information stored in the memory, and the user command is a control signal from a remote control device. Claim 2 An electronic device according to claim 1, wherein the processor identifies the type of the second user voice when the second user voice is received through the microphone within a second threshold time from the time when the operation corresponding to the first user voice is performed, and stores the user response information including information about the operation corresponding to the first user voice and the user command in the memory when the type of the second user voice is a preset type and the user command is received. Claim 3 An electronic device according to paragraph 2, wherein the processor stores in the memory user response information including information on an action corresponding to the first user voice and the user command when the user command is received after the second user voice of the preset type is received. Claim 4 In paragraph 2, the second user voice of the aforementioned preset type is an electronic device that is a user voice containing words expressing negative emotions. Claim 5 An electronic device according to claim 1, wherein the processor controls the display to display content corresponding to the second user voice and to display the guide information overlaid on the content. Claim 6 In paragraph 5, the processor is an electronic device that stops playback of the content while the guide information is displayed. Claim 7 An electronic device according to claim 1, wherein the processor performs an operation corresponding to the user command when the second user voice is received while the number of user response information stored in the memory is greater than or equal to a threshold number. Claim 8 An electronic device according to claim 1, wherein the guide information comprises at least one of a message that induces an action corresponding to the user command or an item for executing an action corresponding to the user command. Claim 9 An electronic device according to claim 1, wherein the processor deletes the user response information corresponding to the first user voice from the memory if the user command is not received through the input unit within a third threshold time after the guide information is provided. Claim 10 An electronic device according to claim 1, further comprising a communication interface; wherein the processor controls the communication interface to transmit information regarding an action corresponding to the first user voice and the user command to a server when the user command is received, receives the user response information from the server through the communication interface, and stores the received user response information in the memory. Claim 11 An electronic device according to claim 1, wherein information regarding an operation corresponding to the first user voice includes information regarding the type of operation to be performed by the electronic device based on the first user voice. Claim 12 A control method for an electronic device comprises: a step of performing an operation corresponding to the first user voice when a first user voice is received; a step of storing user response information including information on the operation corresponding to the first user voice and the user command when a user command is received within a first threshold time from the time when the first user voice is received or the time when the operation corresponding to the first user voice is performed; a step of performing an operation corresponding to the second user voice when a second user voice is received; and a step of providing guide information that induces an operation according to the user command based on the user response information when the type of the operation corresponding to the first user voice and the type of the operation corresponding to the second user voice are the same, wherein the user command is a control signal from a remote control device. Claim 13 A control method according to claim 12, further comprising the step of identifying the type of the second user voice when the second user voice is received within a second threshold time from the time when the action corresponding to the first user voice is performed; wherein the storing step comprises storing the user response information including information about the action corresponding to the first user voice and the user command when the type of the second user voice is a preset type and the user command is received. Claim 14 In claim 13, the above-mentioned storing step is a control method that stores user response information including information on an action corresponding to the first user voice and the user command when the user command is received after the second user voice of the above-mentioned preset type is received. Claim 15 In paragraph 13, the control method wherein the second user voice of the above-mentioned preset type is a user voice containing words expressing negative emotions. Claim 16 A control method according to claim 12, wherein the step of performing an operation corresponding to the second user voice is to display content corresponding to the second user voice, and the step of providing is to display guide information overlaid on the content. Claim 17 In claim 16, the step provided above is a control method that stops the playback of the content while the guide information is displayed. Claim 18 In claim 12, the step provided above is a control method that performs an action corresponding to the user command when the second user voice is received while the number of times the user response information is stored is greater than or equal to a threshold number. Claim 19 A control method according to claim 12, wherein the guide information comprises at least one of a message that induces an action corresponding to the user command or an item for executing an action corresponding to the user command. Claim 20 A control method according to claim 12, further comprising the step of deleting the user response information corresponding to the first user voice if the user command is not received within a third threshold time after the guide information is provided.

Citation Information

Patent Citations

  • electronic apparatus, method for controlling thereof and the computer readable recording medium

    KR1020190065821A

  • Systems and methods for monitoring motion sensor signals and adjusting interaction modes

    US20110279359A1

  • Method of and system for real time feedback in an incremental speech input interface

    US20190051317A1

  • Growth type interactive device

    JP2019101291A

  • Electronic apparatus, method for determining user utterance intention of thereof, and non-transitory computer readable recording medium

    KR1020180071931A