Display device and operating method thereof

Abstract

The present disclosure is intended to assist a voice recognition function when voice recognition fails. A display device according to an embodiment of the present disclosure may include a display, a user input interface that receives a command from a remote control device, and a controller that displays a keyboard window including one or more queries and keyboard items on the display when voice recognition fails for a voice command spoken by a user while the voice recognition function is activated.

Description

Technology Field

[0001] The present disclosure relates to a display device, and more specifically, to a display device capable of providing a voice recognition function. Background Technology

[0002] Digital TV services using wired or wireless communication networks are becoming commonplace. Digital TV services can provide a variety of services that were not available through existing analog broadcasting services.

[0003] For example, digital TV services such as IPTV (Internet Protocol Television) and smart TV provide interactivity that allows users to actively select the types of programs and viewing time. Based on this interactivity, IPTV and smart TV services can also provide various additional services, such as internet search, home shopping, and online games.

[0004] Recent TVs can provide voice recognition capabilities based on user speech. However, conventionally, users often do not know the speech supported by the TV at the time of using the voice recognition function.

[0005] In other words, there are many cases where users do not speak even when the voice recognition function is activated, such as when they press the microphone button on the remote control to use the voice recognition feature.

[0006] As a result, a problem arises where the usability of the TV's voice recognition function is reduced. The problem to be solved

[0007] The present disclosure aims to provide a display device that can enhance the usability of a voice recognition function by using a virtual keyboard window as an input means to assist the voice recognition function.

[0008] The purpose of the present disclosure is to enhance the usability of the voice recognition function by providing multiple recommended queries in the keyboard window when voice recognition fails due to the absence of speech.

[0009] The purpose of the present disclosure is to enhance the usability of a voice recognition function by providing queries similar to the utterances in a keyboard window when voice recognition for multiple utterances fails. means of solving the problem

[0010] A display device according to an embodiment of the present disclosure may include a display, a user input interface that receives commands from a remote control device, and a controller that displays a keyboard window including one or more queries and keyboard items on the display when voice recognition of a voice command spoken by a user fails while a voice recognition function is activated.

[0011] If the controller does not receive the voice command within a certain period of time, it determines that the voice recognition has failed and may display the keyboard window containing a plurality of recommended queries.

[0012] The above controller may determine that the voice recognition has failed if the voice command is recognized within a certain period of time and voice recognition is not possible for a certain number of utterances, and may display the keyboard window containing a plurality of candidate queries. Effects of the invention

[0013] According to an embodiment of the present disclosure, the usability of the voice recognition function is enhanced, thereby providing the user with an improved voice recognition experience.

[0014] According to an embodiment of the present disclosure, a user-customized voice recognition function can be provided by providing recommended queries suitable for the user through a keyboard window.

[0015] According to an embodiment of the present disclosure, a voice recognition service that meets the user's needs can be provided by providing queries similar to commands spoken by the user. Brief explanation of the drawing

[0016] FIG. 1 is a block diagram illustrating the configuration of a display device according to one embodiment of the present invention. FIG. 2 is a block diagram of a remote control device according to one embodiment of the present invention. FIG. 3 shows an example of the actual configuration of a remote control device according to one embodiment of the present invention. Figure 4 shows an example of utilizing a remote control device according to an embodiment of the present invention. FIG. 5 is a flowchart for explaining the operation method of a display device according to one embodiment of the present disclosure. FIG. 6 is a drawing illustrating a setting menu for setting keyboard window exposure according to one embodiment of the present disclosure. FIGS. 7 and FIGS. 8 are drawings illustrating a method for exposing a keyboard window according to an embodiment of the present disclosure. FIG. 9 is a drawing illustrating the configuration of a controller according to one embodiment of the present disclosure. FIG. 10 is a diagram illustrating a method for providing a recommendation query when a voice command is not received within a certain period of time according to an embodiment of the present disclosure. FIG. 11 is a diagram illustrating a method for providing candidate queries when voice recognition fails for a certain number of utterances according to one embodiment of the present disclosure. FIG. 12 is a flowchart illustrating a method of operation of a display device according to another embodiment of the present disclosure. FIG. 13 is a drawing illustrating a recommendation query window that provides a recommendation query based on speech history according to one embodiment of the present disclosure. FIG. 14 is a drawing illustrating a voice help window according to one embodiment of the present disclosure. Specific details for implementing the invention

[0017] Hereinafter, embodiments related to the present invention will be described in more detail with reference to the drawings. The suffixes "module" and "part" for components used in the following description are assigned or used interchangeably solely for the ease of drafting the specification, and do not inherently have distinct meanings or roles.

[0018] A display device according to an embodiment of the present invention is, for example, an intelligent display device that adds computer support functions to a broadcast reception function. While faithful to the broadcast reception function, it also adds internet functions, and can be equipped with a more user-friendly interface such as a handwriting input device, a touch screen, or a spatial remote control. Furthermore, by supporting wired or wireless internet functions, it can be connected to the internet and a computer, and can perform functions such as email, web browsing, banking, or games. A standardized general-purpose OS may be used for these various functions.

[0019] Accordingly, the display device described in the present invention allows various applications to be freely added or removed, for example, on a general-purpose OS kernel, thereby enabling various user-friendly functions to be performed. More specifically, the display device may be, for example, a network TV, HBB TV, smart TV, LED TV, OLED TV, etc., and may also be applicable to smartphones depending on the circumstances.

[0020] FIG. 1 is a block diagram illustrating the configuration of a display device according to one embodiment of the present invention.

[0021] Referring to FIG. 1, the display device (100) may include a broadcast receiver (130), an external device interface (135), a memory (140), a user input interface (150), a controller (170), a wireless communication interface (173), a display (180), a speaker (185), and a power supply circuit (190).

[0022] The broadcast receiver (130) may include a tuner (131), a demodulator (132), and a network interface (133).

[0023] The tuner (131) can tune to a specific broadcast channel according to a channel tuning command. The tuner (131) can receive a broadcast signal for the tuned specific broadcast channel.

[0024] The demodulator (132) can separate the received broadcast signal into a video signal, an audio signal, and a data signal related to the broadcast program, and can restore the separated video signal, audio signal, and data signal into a form that can be output.

[0025] The external device interface (135) can receive an application or a list of applications within an adjacent external device and transmit it to the controller (170) or memory (140).

[0026] The external device interface (135) can provide a connection path between the display device (100) and an external device. The external device interface (135) can receive one or more of video and audio output from an external device connected to the display device (100) wirelessly or via a wired connection, and transmit them to the controller (170). The external device interface (135) may include a plurality of external input terminals. The plurality of external input terminals may include an RGB terminal, one or more HDMI (High Definition Multimedia Interface) terminals, and a component terminal.

[0027] The video signal of an external device input through the external device interface (135) can be output through the display (180). The audio signal of an external device input through the external device interface (135) can be output through the speaker (185).

[0028] The external device that can be connected to the external device interface (135) may be any one of a set-top box, Blu-ray player, DVD player, game console, soundbar, smartphone, PC, USB memory, or home theater, but this is merely an example.

[0029] The network interface (133) may provide an interface for connecting the display device (100) to a wired / wireless network including the Internet network. The network interface (133) may transmit or receive data to or from other users or other electronic devices through the connected network or another network linked to the connected network.

[0030] In addition, some of the content data stored in the display device (100) can be transmitted to another user or other electronic device selected among other users or other electronic devices that are previously registered in the display device (100).

[0031] The network interface (133) can access a specific web page through a connected network or another network linked to the connected network. That is, it can access a specific web page through a network and transmit or receive data with the corresponding server.

[0032] Additionally, the network interface (133) can receive content or data provided by a content provider or network operator. That is, the network interface (133) can receive content such as movies, advertisements, games, VOD, broadcast signals, and related information provided by a content provider or network provider through a network.

[0033] Additionally, the network interface (133) can receive firmware update information and update files provided by the network operator, and can transmit data to the internet or a content provider or network operator.

[0034] The network interface (133) can select and receive a desired application among the applications that are open to the public through the network.

[0035] The memory (140) can store programs for each signal processing and control within the controller (170), and can store signal-processed video, audio, or data signals.

[0036] Additionally, the memory (140) may perform the function of temporarily storing video, audio, or data signals input from an external device interface (135) or a network interface (133), and may also store information regarding a predetermined image through a channel memory function.

[0037] The memory (140) can store an application or a list of applications input from an external device interface (135) or a network interface (133).

[0038] The display device (100) can play content files (video files, still image files, music files, document files, application files, etc.) stored in memory (140) and provide them to the user.

[0039] The user input interface (150) can transmit a signal input by the user to the controller (170) or transmit a signal from the controller (170) to the user. For example, the user input interface (150) can receive and process control signals such as power on / off, channel selection, and screen setting from the remote control device (200) according to various communication methods such as Bluetooth, Ultra Wideband (WB), ZigBee, Radio Frequency (RF) communication, or Infrared (IR) communication, or process to transmit control signals from the controller (170) to the remote control device (200).

[0040] Additionally, the user input interface (150) can transmit control signals input from local keys (not shown), such as power key, channel key, volume key, and setting value, to the controller (170).

[0041] The image signal processed by the controller (170) can be input to the display (180) and displayed as an image corresponding to the image signal. Additionally, the image signal processed by the controller (170) can be input to an external output device through the external device interface (135).

[0042] The voice signal processed by the controller (170) can be output as audio to the speaker (185). Additionally, the voice signal processed by the controller (170) can be input to an external output device through the external device interface (135).

[0043] In addition, the controller (170) can control the overall operation within the display device (100).

[0044] Additionally, the controller (170) can control the display device (100) by means of user commands or internal programs entered through the user input interface (150), and can connect to a network to allow the user to download an application or a list of applications desired by the user into the display device (100).

[0045] The controller (170) enables the processed video or audio signal, such as channel information selected by the user, to be output through the display (180) or speaker (185).

[0046] Additionally, the controller (170) enables a video signal or audio signal from an external device, such as a camera or camcorder, input through the external device interface (135) according to an external device video playback command received through the user input interface (150), to be output through the display (180) or speaker (185).

[0047] Meanwhile, the controller (170) can control the display (180) to display video, for example, broadcast video input through the tuner (131), external input video input through the external device interface (135), video input through the network interface, or video stored in the memory (140) can be controlled to be displayed on the display (180). In this case, the video displayed on the display (180) may be a still image or a video, and may be a 2D image or a 3D image.

[0048] Additionally, the controller (170) can control the playback of content stored in the display device (100), received broadcast content, or external input content input from the outside, and the content may be in various forms such as broadcast video, external input video, audio file, still image, connected web screen, and document file.

[0049] The wireless communication interface (173) can communicate with an external device via wired or wireless communication. The wireless communication interface (173) can perform short-range communication with an external device. To this end, the wireless communication interface (173) can support short-range communication by using at least one of Bluetooth™, RFID (Radio Frequency Identification), Infrared Data Association (IrDA), Ultra Wideband (UWB), ZigBee, Near Field Communication (NFC), Wireless-Fidelity (Wi-Fi), Wi-Fi Direct, and Wireless Universal Serial Bus (Wireless USB) technologies. Such wireless communication interface (173) can support wireless communication between the display device (100) and a wireless communication system, between the display device (100) and another display device (100), or between the display device (100) and a network where the display device (100, or an external server) is located, through wireless area networks. Short-range wireless communication networks can be Wireless Personal Area Networks.

[0050] Here, another display device (100) may be a mobile terminal such as a wearable device (e.g., a smartwatch, smart glass, HMD (head mounted display)), a smartphone, etc., capable of exchanging data with (or interacting with) the display device (100) according to the present invention. The wireless communication interface (173) may detect (or recognize) a wearable device capable of communicating around the display device (100).

[0051] Furthermore, if the detected wearable device is a device authenticated to communicate with the display device (100) according to the present invention, the controller (170) can transmit at least a portion of the data processed in the display device (100) to the wearable device through the wireless communication interface (173). Accordingly, the user of the wearable device can use the data processed in the display device (100) through the wearable device.

[0052] The display (180) can generate a driving signal by converting the video signal, data signal, OSD signal processed by the controller (170) or the video signal, data signal, etc. received from the external device interface (135) into R, G, and B signals, respectively.

[0053] Meanwhile, since the display device (100) illustrated in FIG. 1 is merely an embodiment of the present invention, some of the illustrated components may be integrated, added, or omitted depending on the specifications of the actual implemented display device (100).

[0054] That is, as needed, two or more components may be combined into a single component, or a single component may be subdivided into two or more components. In addition, the functions performed in each block are intended to explain embodiments of the present invention, and the specific operations or devices thereof do not limit the scope of the present invention.

[0055] According to another embodiment of the present invention, the display device (100) may receive and play video through a network interface (133) or an external device interface (135) without having a tuner (131) and a demodulator (132), unlike as shown in FIG. 1.

[0056] For example, the display device (100) may be implemented by separating it into a video processing device, such as a set-top box, for receiving broadcast signals or content according to various network services, and a content playback device for playing content input from the video processing device.

[0057] In this case, the operation method of the display device according to the embodiment of the present invention described below may be performed by any one of the display device (100) described with reference to FIG. 1, as well as an image processing device such as the separated set-top box, or a content playback device having a display (180) and an audio output unit (185).

[0058] Next, with reference to FIGS. 2 and 3, a remote control device according to an embodiment of the present invention will be described.

[0059] FIG. 2 is a block diagram of a remote control device according to one embodiment of the present invention, and FIG. 3 shows an example of the actual configuration of a remote control device (200) according to one embodiment of the present invention.

[0060] First, referring to FIG. 2, the remote control device (200) may include a fingerprint reader (210), a wireless communication circuit (220), a user input interface (230), a sensor (240), an output interface (250), a power supply circuit (260), a memory (270), a controller (280), and a microphone (290).

[0061] Referring to FIG. 2, the wireless communication circuit (220) transmits and receives signals with any one of the display devices according to the embodiments of the present invention described above.

[0062] The remote control device (200) is equipped with an RF circuit (221) capable of transmitting and receiving signals to and from the display device (100) according to RF communication standards, and may be equipped with an IR circuit (223) capable of transmitting and receiving signals to and from the display device (100) according to IR communication standards. Additionally, the remote control device (200) may be equipped with a Bluetooth circuit (225) capable of transmitting and receiving signals to and from the display device (100) according to Bluetooth communication standards. Furthermore, the remote control device (200) may be equipped with an NFC circuit (227) capable of transmitting and receiving signals to and from the display device (100) according to NFC (Near Field Communication) communication standards, and may be equipped with a WLAN circuit (229) capable of transmitting and receiving signals to and from the display device (100) according to WLAN (Wireless LAN) communication standards.

[0063] Additionally, the remote control device (200) transmits a signal containing information regarding the movement of the remote control device (200), etc., to the display device (100) through the wireless communication circuit (220).

[0064] Meanwhile, the remote control device (200) can receive a signal transmitted by the display device (100) through the RF circuit (221), and, if necessary, can transmit commands regarding power on / off, channel change, volume change, etc. to the display device (100) through the IR circuit (223).

[0065] The user input interface (230) may be composed of a keypad, buttons, a touchpad, or a touch screen. The user can input commands related to the display device (100) to the remote control device (200) by operating the user input interface (230). If the user input interface (230) is equipped with a hard key button, the user can input commands related to the display device (100) to the remote control device (200) through a push operation of the hard key button. This will be explained with reference to FIG. 3.

[0066] Referring to FIG. 3, the remote control device (200) may include a plurality of buttons. The plurality of buttons may include a fingerprint recognition button (212), a power button (231), a home button (232), a live button (233), an external input button (234), a volume control button (235), a voice recognition button (236), a channel change button (237), a confirmation button (238), and a back button (239).

[0067] The fingerprint recognition button (212) may be a button for recognizing a user's fingerprint. In one embodiment, the fingerprint recognition button (212) may be capable of a push operation and may receive a push operation and a fingerprint recognition operation.

[0068] The power button (231) may be a button for turning the power of the display device (100) on / off.

[0069] The home button (232) may be a button for moving to the home screen of the display device (100).

[0070] The live button (233) may be a button for displaying a live broadcast program.

[0071] The external input button (234) may be a button for receiving an external input connected to the display device (100).

[0072] The volume control button (235) may be a button for adjusting the volume output by the display device (100).

[0073] The voice recognition button (236) may be a button for receiving the user's voice and recognizing the received voice.

[0074] The channel change button (237) may be a button for receiving a broadcast signal of a specific broadcast channel.

[0075] The confirmation button (238) may be a button for selecting a specific function, and the back button (239) may be a button for returning to the previous screen.

[0076] Figure 2 is explained again.

[0077] If the user input interface (230) is equipped with a touchscreen, the user can input commands related to the display device (100) to the remote control device (200) by touching the soft keys on the touchscreen. Additionally, the user input interface (230) may be equipped with various types of input means that the user can operate, such as scroll keys or jog keys, and this embodiment does not limit the scope of the present invention.

[0078] The sensor (240) may be equipped with a gyroscope sensor (241) or an accelerometer sensor (243), and the gyroscope sensor (241) may sense information regarding the movement of the remote control device (200).

[0079] For example, the gyroscope sensor (241) can sense information regarding the operation of the remote control device (200) based on the x, y, and z axes, and the accelerometer sensor (243) can sense information regarding the movement speed of the remote control device (200). Meanwhile, the remote control device (200) may further be equipped with a distance measuring sensor to sense the distance to the display (180) of the display device (100).

[0080] The output interface (250) can output a video or audio signal corresponding to the operation of the user input interface (230) or a signal transmitted from the display device (100).

[0081] The user can recognize whether the output interface (250) is operated by the user input interface (230) or whether the display device (100) is controlled.

[0082] For example, the output interface (250) may be equipped with an LED (251) that lights up when the user input interface (230) is operated or when a signal is transmitted or received with the display device (100) through the wireless communication unit (225), a vibrator (253) that generates vibration, a speaker (255) that outputs sound, or a display (257) that outputs video.

[0083] In addition, the power supply circuit (260) supplies power to the remote control device (200), and can reduce power waste by stopping the power supply when the remote control device (200) does not move for a predetermined period of time.

[0084] The power supply circuit (260) can resume power supply when a predetermined key provided in the remote control device (200) is operated.

[0085] The memory (270) can store various types of programs, application data, etc. required for the control or operation of the remote control device (200).

[0086] When the remote control device (200) transmits and receives signals wirelessly through the RF circuit (221) and the display device (100), the remote control device (200) and the display device (100) transmit and receive signals through a predetermined frequency band.

[0087] The controller (280) of the remote control device (200) can store and refer to information regarding frequency bands, etc., that can wirelessly transmit and receive signals with the display device (100) paired with the remote control device (200) in the memory (270).

[0088] The controller (280) controls all matters related to the control of the remote control device (200). The controller (280) can transmit a signal corresponding to a predetermined key operation of the user input interface (230) or a signal corresponding to the movement of the remote control device (200) sensed by the sensor (240) to the display device (100) through the wireless communication unit (225).

[0089] In addition, the microphone (290) of the remote control device (200) can acquire voice.

[0090] The microphone (290) may be provided in multiple numbers.

[0091] Next, Figure 4 is explained.

[0092] Figure 4 shows an example of utilizing a remote control device according to an embodiment of the present invention.

[0093] FIG. 4(a) illustrates a pointer (205) corresponding to a remote control device (200) being displayed on a display (180).

[0094] The user can move or rotate the remote control device (200) up and down, left and right. The pointer (205) displayed on the display (180) of the display device (100) corresponds to the movement of the remote control device (200). Since the pointer (205) of the remote control device (200) moves and is displayed according to the movement in 3D space as shown in the drawing, it can be named a spatial remote control.

[0095] FIG. 4(b) illustrates that when the user moves the remote control device (200) to the left, the pointer (205) displayed on the display (180) of the display device (100) also moves to the left in response.

[0096] Information regarding the movement of the remote control device (200) detected through the sensor of the remote control device (200) is transmitted to the display device (100). The display device (100) can calculate the coordinates of the pointer (205) from the information regarding the movement of the remote control device (200). The display device (100) can display the pointer (205) to correspond to the calculated coordinates.

[0097] FIG. 4(c) illustrates a case where, while pressing a specific button within the remote control device (200), the user moves the remote control device (200) away from the display (180). By doing so, the selected area within the display (180) corresponding to the pointer (205) can be zoomed in and enlarged.

[0098] Conversely, when the user moves the remote control device (200) closer to the display (180), the selected area within the display (180) corresponding to the pointer (205) can be zoomed out and reduced in size.

[0099] Meanwhile, when the remote control device (200) moves away from the display (180), the selected area may be zoomed out, and when the remote control device (200) moves closer to the display (180), the selected area may be zoomed in.

[0100] Additionally, when a specific button within the remote control device (200) is pressed, recognition of up-down and left-right movement may be excluded. That is, when the remote control device (200) moves away from or closer to the display (180), up-down, left-right movement is not recognized, and only forward-backward movement is recognized. When the specific button within the remote control device (200) is not pressed, only the pointer (205) moves according to the up-down, left-right movement of the remote control device (200).

[0101] Meanwhile, the movement speed or direction of movement of the pointer (205) can correspond to the movement speed or direction of movement of the remote control device (200).

[0102] Meanwhile, the pointer in this specification refers to an object displayed on the display (180) in response to the operation of the remote control device (200). Accordingly, the pointer (205) can be an object of various shapes other than the arrow shape shown in the drawing. For example, it may be a concept including a point, a cursor, a prompt, a thick outline, etc. Furthermore, the pointer (205) can be displayed corresponding to either a point on the horizontal axis or a vertical axis on the display (180), and it is also possible to display it corresponding to multiple points, such as a line or a surface.

[0103] FIG. 5 is a flowchart for explaining the operation method of a display device according to one embodiment of the present disclosure.

[0104] FIG. 5 is described assuming that the source of the utterance is a remote control device (200), but it may also be applicable if the source of the utterance is a user. The source of the utterance may represent a provider who provided the voice command received by the display device (100).

[0105] Referring to Fig. 5, The controller (170) of the display device (100) enables the voice recognition function (S501).

[0106] The voice recognition function may be a function that recognizes the results of intent analysis for a voice command spoken by a user.

[0107] In one embodiment, the controller (170) may receive a command to activate a voice recognition function from the remote control device (200). For example, the user may select a microphone button (not shown) provided on the remote control device (200). The remote control device (200) may transmit a control command to the display device (100) to activate the voice recognition function based on the selection of the microphone button.

[0108] The display device (100) can activate a voice recognition function according to a control command received from a remote control device (200).

[0109] In another embodiment, the controller (170) can activate the voice recognition function by recognizing a command word spoken by the user. The controller (170) can receive the command word through a microphone (290) provided in the display device (100) and activate the voice recognition function according to the received command word.

[0110] The controller (170) determines whether a voice command has been received within a certain period of time while the voice recognition function is activated (S503).

[0111] The time limit may be 5 seconds, but this is merely an example figure.

[0112] If the controller (170) does not receive a voice command within a certain period of time, it determines whether the setting for displaying the keyboard window is turned on (S505).

[0113] In one embodiment, the keyboard window may be a window that is displayed to assist the voice recognition function when the user's speech command is not recognized.

[0114] The keyboard window may include one or more recommendation queries to assist the user's voice recognition function when voice recognition based on the user's utterance fails.

[0115] Recommended queries can be recommended keywords.

[0116] Users can turn the setting for keyboard window display on or off through the menu.

[0117] This will be explained with reference to Fig. 6.

[0118] FIG. 6 is a drawing illustrating a setting menu for setting keyboard window exposure according to one embodiment of the present disclosure.

[0119] Referring to FIG. 6, a settings menu (600) for displaying the keyboard window is shown.

[0120] The controller (170) can display the settings menu (600) on the display (180) upon receiving a command for displaying the menu.

[0121] The settings menu (600) may be a menu for setting whether to display the keyboard window in the event of a voice recognition failure. The user can turn the keyboard window display setting on or off through a toggle button (610) included in the settings menu (600).

[0122] The settings menu (600) is necessary because there may be users who do not want the keyboard window to be displayed even if voice recognition fails.

[0123] Again, Figure 5 is explained.

[0124] When the controller (170) is turned on for keyboard window exposure, it displays the keyboard window and recommended query on the display (180) (S507).

[0125] In one embodiment, when a content image is being played on the display (180), the controller (170) can display a keyboard window superimposed on the content image.

[0126] In one embodiment, the controller (170) can obtain a recommendation query related to an app currently running on the display device (100) and display the obtained recommendation query in a keyboard window.

[0127] In another embodiment, the controller (170) can obtain a recommendation query that can control the function of the display device (100) and display the obtained recommendation query in the keyboard window.

[0128] The keyboard window will be explained with reference to FIGS. 7 and FIGS. 8.

[0129] FIGS. 7 and FIGS. 8 are drawings illustrating a method for exposing a keyboard window according to an embodiment of the present disclosure.

[0130] Referring to FIG. 7, the display device (100) can activate a voice recognition function while playing a content video (700). As the voice recognition function is activated, the display device (100) can display a guide window (701) that guides voice recognition.

[0131] The guide window (701) is a window that induces specific utterances and may include multiple voice commands.

[0132] The display device (100) can identify that voice recognition fails during the playback of the content video (700) and display the first keyboard window (710).

[0133] In one embodiment, a case where voice recognition fails may be a case where a voice command is not received within a certain period of time while the voice recognition function is activated.

[0134] In another embodiment, a case where voice recognition fails may be a case where voice recognition is impossible for more than a certain number of utterances while the voice recognition function is activated.

[0135] The first keyboard window (710) can be displayed superimposed on the content image (700).

[0136] The first keyboard window (710) may be a window representing a virtual keyboard.

[0137] The first keyboard window (710) may include a plurality of recommended queries (711, 712, 713, 714), a query input box (715), a voice search button (716), and a keyboard item (717) including a plurality of character keys.

[0138] In one embodiment, each of the plurality of recommendation queries (711, 712, 713, 714) may be a keyword related to an app that plays the content video (700).

[0139] In another embodiment, each of the plurality of recommendation queries (711, 712, 713, 714) may be a keyword related to the function control of the display device (100).

[0140] In one embodiment, the query input box (715) may be a window that displays a query selected according to user input among a plurality of recommended queries (711, 712, 713, 714).

[0141] In another embodiment, the query input box (715) may be a window for displaying text selected via a keyboard item (717).

[0142] The voice search button (716) may be a button for executing a search for a selected query among a plurality of recommended queries (711, 712, 713, 714) or a query entered in the query input box (715).

[0143] The display device (100) can perform a search for a recommended query as the voice search button (716) is selected and display the search results on the display (180).

[0144] In one embodiment, the display device (100) can transmit text for a recommendation query to a Natural Language Process (NLP) server and receive search results for the recommendation query from the NLP server.

[0145] In another embodiment, the display device (100) can perform a search for recommendation queries through its own integrated search app.

[0146] In another embodiment, the display device (100) can perform a search for a recommendation query through the running app and provide the search results through the app, if the currently running app has a search function.

[0147] Next, Figure 8 is explained.

[0148] Referring to FIG. 7, the display device (100) can activate a voice recognition function while playing a content video (700). As the voice recognition function is activated, the display device (100) can display a guide window (701) that guides voice recognition.

[0149] The display device (100) can identify that voice recognition fails during the playback of the content video (700) and display a second keyboard window (800).

[0150] The second keyboard window (800) may be a different type of window from the first keyboard window (710).

[0151] The display device (100) can display a window set according to the setting menu among the first keyboard window (710) and the second keyboard window (800).

[0152] The second keyboard window (800) can be displayed superimposed on the content image (700).

[0153] The second keyboard window (800) may be a window representing a virtual keyboard.

[0154] The second keyboard window (800) may include a left arrow key (811), a right arrow key (812), a query input box (813), a voice search button (815), and a keyboard item (816).

[0155] The query input box (813) may include a query display area (814) that displays one recommended query.

[0156] The left arrow key (811) and the right arrow key (812) may be keys for changing recommended queries displayed in the query display area (814). The user can change the recommended queries displayed in the query display area (814) by selecting the left arrow key (811) or the right arrow key (812).

[0157] In one embodiment, the recommended query displayed in the query display area (814) may be a keyword related to the app playing the content video (700).

[0158] In another embodiment, the recommended query displayed in the query display area (814) may be a keyword related to the function control of the display device (100).

[0159] The query input box (813) may be a window for displaying text selected via a keyboard item (816).

[0160] The voice search button (815) may be a button for executing a search for a recommended query displayed in the query display area (814) or a query entered in the query input box (813).

[0161] The display device (100) can perform a search for a recommended query as the voice search button (815) is selected and display the search results on the display (180).

[0162] Accordingly, according to an embodiment of the present disclosure, if the voice recognition function cannot be used normally, a keyboard window for voice recognition integration may be displayed. Accordingly, the user may be assisted in using the voice recognition function, thereby improving the usability of the voice recognition function.

[0163] Again, Figure 5 is explained.

[0164] When the controller (170) is off the setting for displaying the keyboard window, it displays a notification on the display (180) indicating that voice recognition is not possible (S509).

[0165] If the display of the keyboard window is turned off through the settings menu (600), the controller (170) may not display the keyboard window even if voice recognition fails. That is, the controller (170) may display a notification on the display (180) indicating that voice recognition is impossible or has failed.

[0166] Meanwhile, if the controller (170) receives a voice command within a certain period of time, it determines whether voice recognition is impossible for a certain number of utterances (S511).

[0167] In one embodiment, the controller (170) receives m voice commands for n seconds, and if m voice recognitions fail, it can determine that voice recognition has failed.

[0168] The controller (170) sends m voice commands to the NLP server, and if it does not receive any analysis results for the voice commands from the NLP server, it can determine that voice recognition has failed.

[0169] When the controller (170) determines that voice recognition is not possible for a certain number of utterances, it determines whether the setting for displaying the keyboard window is turned on (S505), and then performs the subsequent steps.

[0170] When voice recognition is not possible for a certain number of utterances, the controller (170) displays the analysis result information for the received voice command on the display (180) (S513).

[0171] The controller (170) can transmit a voice signal corresponding to a voice command to a STT (Speech To Text) server.

[0172] The STT server can convert speech signals into text and send the converted text to an NLP server.

[0173] The NLP server can generate analysis result information indicating the result of performing intent analysis on text and transmit the generated analysis result information to a display device (100). The analysis result information may include search results for text.

[0174] The NLP server may transmit analysis result information to a search server, receive search results from the search server, and transmit the received search results to a display device (100).

[0175] FIG. 9 is a drawing illustrating the configuration of a controller according to one embodiment of the present disclosure.

[0176] Referring to FIG. 9, the controller (170) may include an NLP control unit (910), a keyboard setting control unit (920), a recommendation query control unit (930), a keyboard window output unit (940), an STT control unit (950), and a speech history management unit (960).

[0177] The NLP control unit (910) can receive a command to release the microphone button provided in the remote control device (200) from the remote control device (200).

[0178] The NLP control unit (910) can request keyboard window exposure setting information from the keyboard setting control unit (920) upon detecting that it has not received a voice command within a certain period of time, and receive a response of keyboard window exposure setting information in response to the request.

[0179] When the keyboard window exposure setting is turned on, the NLP control unit (910) can request a recommendation query from the recommendation query control unit (930) and receive a recommendation query response in response to the request for a recommendation query.

[0180] The keyboard setting control unit (920) receives user input for whether to set the keyboard window exposure, and can turn the setting of the keyboard window exposure on or off according to the received user input.

[0181] The recommendation query control unit (930) can generate one or more recommendation queries.

[0182] The recommendation query control unit (930) may be a component provided in the NLP server. In this case, the NLP control unit (910) may request a recommendation query from the NLP server through the network interface (133) and receive a recommendation query from the NLP server.

[0183] The keyboard window output unit (940) receives a selection command for a recommendation query and a voice search command through the keyboard window, and can transmit a search request for the selected recommendation query to the NLP control unit (910) according to the received voice search command.

[0184] The STT control unit (950) can convert voice commands into text and obtain keywords most similar to the converted text. The STT control unit (950) may be a component provided in an NLP server. In this case, the NLP control unit (910) can request candidate queries from the NLP server through the network interface (133) and receive candidate queries from the NLP server.

[0185] The utterance history management unit (960) can store or retrieve utterance history information corresponding to the user's identification information.

[0186] FIG. 10 is a diagram illustrating a method for providing a recommendation query when a voice command is not received within a certain period of time according to an embodiment of the present disclosure.

[0187] Referring to Fig. 10, The NLP control unit (910) receives a command to release the microphone button provided in the remote control device (200) from the remote control device (200) (S1001).

[0188] A release command for the microphone button can be generated when the microphone button provided in the remote control device (200) is pressed and then released.

[0189] The NLP control unit (910) can activate the voice recognition function of the display device (100) upon receiving a release command from the microphone button.

[0190] The NLP control unit (910) detects that it has not received a voice command within a certain period of time (S1003).

[0191] The NLP control unit (910) can detect that a voice command is not received within a certain period of time while the voice recognition function is activated.

[0192] The NLP control unit (910) detects that it has not received a voice command within a certain period of time and requests keyboard window exposure setting information from the keyboard setting control unit (920) (S1005), and receives a response of keyboard window exposure setting information in response to the request (S1007).

[0193] The response regarding keyboard window display settings information is <true>value or <false>It may include any one of the values.

[0194] <true>The value can indicate that the keyboard window exposure setting is turned on in the settings menu (600), and <false>The value may indicate that the keyboard window exposure setting is off in the settings menu (600).

[0195] When the keyboard window exposure setting is turned on, the NLP control unit (910) requests a recommendation query from the recommendation query control unit (930) (S1009), and receives a recommendation query response in response to the request for a recommendation query (S1011).

[0196] The recommendation query control unit (930) can obtain one or more recommendation queries in response to a request for a recommendation query.

[0197] In one embodiment, the recommendation query control unit (930) can generate a recommendation query related to an app currently running on the display device (100).

[0198] In another embodiment, the recommendation query control unit (930) can generate a recommendation query related to the content video currently being played on the display device (100).

[0199] In another embodiment, the recommendation query control unit (930) can generate a recommendation query that is relevant to controlling the function of the display device (100).

[0200] The NLP control unit (910) transmits a keyboard window display request containing a recommendation query to the keyboard window output unit (940) (S1013).

[0201] The keyboard window output unit (940) outputs the keyboard window to the display (180) in response to a request to display the keyboard window (S1015).

[0202] The keyboard window output unit (940) receives a voice search command through the keyboard window (S1017) and transmits a search request for a recommendation query to the NLP control unit (910) according to the received voice search command (S1019).

[0203] The keyboard window output unit (940) can determine that a voice search command has been received when either the voice search button (716) described in FIG. 7 or the voice search button (815) described in FIG. 8 is selected.

[0204] FIG. 11 is a diagram illustrating a method for providing candidate queries when voice recognition fails for a certain number of utterances according to one embodiment of the present disclosure.

[0205] Referring to Fig. 11, The NLP control unit (910) receives a command to release the microphone button provided in the remote control device (200) from the remote control device (200) (S1101).

[0206] A release command for the microphone button can be generated when the microphone button provided in the remote control device (200) is pressed and then released.

[0207] The NLP control unit (910) can activate the voice recognition function of the display device (100) upon receiving a release command from the microphone button.

[0208] The NLP control unit (910) detects that voice recognition fails more than a certain number of utterances (S1103).

[0209] The NLP control unit (910) receives voice commands multiple times, and if it fails to obtain information on the results of intent analysis for the multiple voice commands, it can determine that voice recognition has failed.

[0210] The NLP control unit (910) requests keyboard window exposure setting information from the keyboard setting control unit (920) (S1105), and receives a response of keyboard window exposure setting information in response to the request (S1107).

[0211] The response regarding keyboard window display settings information is <true>value or <false>It may include any one of the values.

[0212] <true>The value can indicate that the keyboard window exposure setting is turned on in the settings menu (600), and <false>The value may indicate that the keyboard window exposure setting is off in the settings menu (600).

[0213] When the keyboard window exposure setting is turned on, the NLP control unit (910) requests a candidate query from the STT control unit (950) (S1109), and receives a candidate query response from the STT control unit (950) in response to the request for a candidate query (S1111).

[0214] A candidate query can be a type of recommended query provided in the keyboard window.

[0215] The STT control unit (950) can obtain one or more candidate queries in response to a request for a candidate query.

[0216] The STT control unit (950) can obtain voice commands that are subject to voice recognition failure and multiple candidate queries with high reliability.

[0217] Specifically, the STT control unit (950) can convert a voice command that is subject to voice recognition failure into text and obtain candidate queries similar to the converted text (with a confidence level greater than or equal to a preset value).

[0218] The NLP control unit (910) transmits a keyboard window exposure request containing candidate queries to the keyboard window output unit (940) (S1113).

[0219] The number of candidate queries can be 3, but this is merely an example figure.

[0220] The keyboard window output unit (940) outputs the keyboard window to the display (180) in response to a request to display the keyboard window (S1115).

[0221] The keyboard window output unit (940) receives a voice search command through the keyboard window (S1117) and transmits a search request for a recommendation query to the NLP control unit (910) according to the received voice search command (S1119).

[0222] The keyboard window output unit (940) can determine that a voice search command has been received when either the voice search button (716) described in FIG. 7 or the voice search button (815) described in FIG. 8 is selected.

[0223] FIG. 12 is a flowchart illustrating a method of operation of a display device according to another embodiment of the present disclosure.

[0224] In particular, Fig. 12 is a diagram illustrating an example of providing recommendation queries in different forms depending on the utterance source.

[0225] In FIG. 12, when the utterance source is a user, voice recognition according to the voice command can be named long-range voice recognition, and when the utterance source is a remote control device (200), voice recognition according to the voice command can be named short-range voice recognition.

[0226] Referring to Fig. 12, The controller (170) of the display device (100) detects a failure of voice recognition (S1201).

[0227] In one embodiment, the controller (170) may detect that voice recognition has failed if it does not receive a voice command within a certain period of time while the voice recognition function is activated.

[0228] In another embodiment, the controller (170) may detect that voice recognition has failed if it has not obtained analysis results for multiple voice commands received while the voice recognition function is enabled.

[0229] The controller (170) determines whether the utterance source that provided the voice command during the voice recognition failure process is the user (S1203).

[0230] If the controller (170) does not receive a voice command from the remote control device (200), it can determine that the utterance source is the user.

[0231] If the controller (170) determines that the utterance source is not a user, the utterance source is a remote control device (200), and performs step S505 of FIG. 5 and subsequent steps.

[0232] When the controller (170) determines that the utterance source is a user, it determines whether the user's confirmation of the utterance recommendation provision has been received (S1205).

[0233] When the controller (170) determines that the utterance source is a user, it can display a notification window on the display (180) indicating whether utterance recommendation can be provided.

[0234] For example, the notification window can include the text "<Would you like a speech recommendation? If so, please say yes>".

[0235] When the controller (170) receives a user’s utterance of <yes>, it can determine that the user’s confirmation of the utterance recommendation has been received.

[0236] If the controller (170) receives a user's utterance of "No," it may determine that the user's confirmation regarding the utterance recommendation provision has not been received. In this case, the controller (170) may display the voice help described below.

[0237] When the controller (170) receives confirmation from the user regarding the provision of speech recommendations, it determines whether the recognition of the user who spoke the voice command was successful (S1207).

[0238] The controller (170) can determine whether user recognition is successful through a camera (not shown). That is, the controller (170) can capture a user image through the camera and identify the user based on the captured image.

[0239] The controller (170) can obtain user identification information based on the image. The user identification information may include one or more of the user's name, age, gender, and ID.

[0240] When user recognition is successful, the controller (170) determines whether there is a speech history of the recognized user (S1209).

[0241] The controller (170) can search for a speech history corresponding to the identification information based on the user's identification information.

[0242] The controller (170) can obtain speech history information corresponding to the user's identification information from the memory (140). The speech history information may include voice commands previously spoken by the user.

[0243] If the controller (170) has a speech history of the recognized user, it displays a recommendation query based on the speech history on the display (180) (S1211).

[0244] The controller (170) can obtain previously spoken voice commands and voice commands similar to previously spoken voice commands as recommendation queries based on speech history information.

[0245] FIG. 13 is a drawing illustrating a recommendation query window that provides a recommendation query based on speech history according to one embodiment of the present disclosure.

[0246] Referring to FIG. 13, if there is a recognized user's speech history, the display device (100) can display a recommendation query window (1300) on the display (180) that includes a plurality of recommendation queries based on the speech history.

[0247] The recommendation query window (1300) may include voice commands (recommendation queries) previously spoken by the recognized user and text indicating the initiation of the voice command speech.

[0248] In the case of long-distance voice recognition, the reason for providing a recommendation query window (1300) instead of a keyboard window is that when the distance between the user and the display device (100) is far, it is difficult to control the keyboard window through the remote control device (200).

[0249] Conversely, in the case of short-range voice recognition, the reason for providing a keyboard window is that when the distance between the user and the display device (100) is short, it is easy to control the keyboard window through the remote control device (200).

[0250] Again, Figure 12 is explained.

[0251] If there is no speech history of the recognized user, the controller (170) determines whether there is a recommendation query based on time information (S1213).

[0252] The controller (170) can obtain recommended queries for each day or time period from the server or memory (140).

[0253] If there is a recommended query based on time information, the controller (170) displays the recommended query based on time information on the display (180) (S1215).

[0254] When the controller (170) receives a recommendation query based on time information from the server or when it is stored in memory (140), it can display the acquired recommendation query on the display (180).

[0255] If there is no recommended query based on time information, the controller (170) displays a voice help window on the display (180) (S1217).

[0256] FIG. 14 is a drawing illustrating a voice help window according to one embodiment of the present disclosure.

[0257] Referring to FIG. 14, if the display device (100) cannot obtain a recommended query based on speech history and time information, it may display a voice help window (1400) on the display (180) to provide help for the voice recognition function.

[0258] According to an embodiment of the present disclosure, the display device (100) may display a first keyboard window (710) according to FIG. 7 or a second keyboard window (800) according to FIG. 8 when the utterance source is a remote control device (200) (near-field voice recognition).

[0259] The display device (100) can display a recommendation query window (1300) according to FIG. 13 when the speech source is a user (far-field voice recognition).

[0260] In this way, the display device (100) can provide recommendation queries in different forms depending on the distance between the utterance source or the user and the display device (100).

[0261] In the embodiments of the present disclosure, long-distance voice recognition or short-distance voice recognition is distinguished depending on whether the speech source is a user or a remote control device (200), but it is not necessary to be limited thereto.

[0262] That is, the display device (100) may be equipped with a separate distance sensor (e.g., an infrared sensor) for measuring the distance between the user and the display device (100), so as to measure the distance between the user and the display device (100).

[0263] The display device (100) can perform short-range voice recognition when the measured distance is within a preset distance, and perform long-range voice recognition when the measured distance is greater than or equal to a preset distance.

[0264] According to one embodiment of the present disclosure, the above-described method can be implemented as code that is readable by a processor on a medium on which a program is recorded. Examples of media that are readable by a processor include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc.

[0265] The display device described above is not limited to the configuration and method of the embodiments described above; rather, all or part of each embodiment may be selectively combined to allow for various modifications to be made.< / false> < / true> < / false> < / true> < / false> < / true> < / false> < / true>

Claims

Claim 1 A display device comprising: a display; a user input interface receiving a command from a remote control device; and a controller that displays a keyboard window including a plurality of recommended keywords and keyboard items on the display when a change in voice recognition occurs for a voice command spoken by a user while a voice recognition function is activated, wherein each of the plurality of recommended keywords is any one of a recommended keyword related to an app running on the display device, a recommended keyword related to a content video playing on the display, and a recommended keyword for controlling the function of the display device, and the keyboard window includes a recommended keyword input box in which a recommended keyword selected based on user input among the plurality of recommended keywords is displayed, and a voice search button for executing a search for the keyword entered in the recommended keyword input box. Claim 2 A display device according to claim 1, wherein if the controller does not receive the voice command within a certain period of time, the controller determines that the voice recognition has failed and displays the keyboard window including a plurality of recommended keywords. Claim 3 delete Claim 4 In claim 1, the controller displays a search result for a selected recommended keyword on the display in response to a command to select a selected recommended keyword among one or more recommended keywords. Claim 5 A display device according to claim 1, wherein the controller recognizes the voice command within a certain time and, if voice recognition is not possible for a certain number of utterances, determines that the voice recognition has failed and displays the keyboard window including a plurality of candidate recommendation keywords. Claim 6 In paragraph 5, each of the above multiple candidate recommendation keywords is a keyword whose reliability with the above voice command is greater than or equal to a preset value, in a display device. Claim 7 delete Claim 8 In claim 1, the controller displays a setting menu for setting the exposure of the keyboard window on the display, and a display device that sets whether the keyboard window is exposed according to a command entered through the setting menu. Claim 9 A display device according to claim 1, wherein the controller displays the keyboard window on the display when the speech source providing the voice command is the remote control device, and displays a recommendation keyword window including a plurality of recommendation keywords on the display when the speech source providing the voice command is the user. Claim 10 In claim 9, the controller is a display device in which the plurality of recommended keywords represent voice commands previously spoken by the user. Claim 11 A method of operation of a display device comprising: receiving a command for activating a voice recognition function; determining whether voice recognition for a voice command spoken by a user fails while the voice recognition function is activated according to the command; and, if voice recognition fails, displaying a keyboard window including a plurality of recommended keywords and keyboard items, wherein each of the plurality of recommended keywords is any one of a recommended keyword related to an app running on the display device, a recommended keyword related to a content video playing on the display, and a recommended keyword for controlling the function of the display device, and the keyboard window comprises a recommended keyword input box in which a recommended keyword selected based on user input among the plurality of recommended keywords is displayed, and a voice search button for executing a search for the keyword entered in the recommended keyword input box. Claim 12 A method of operation of a display device according to claim 11, wherein if the voice command is not received within a certain period of time, the voice recognition is determined to have failed, and the keyboard window including a plurality of recommended keywords is displayed. Claim 13 delete Claim 14 A method of operation of a display device that displays search results for a selected recommended keyword on the display in response to a command to select a selected recommended keyword among one or more recommended keywords in claim 11. Claim 15 A method of operation of a display device according to claim 11, wherein if the voice command is recognized within a certain time and voice recognition is not possible for a certain number of utterances, the voice recognition is determined to have failed and the keyboard window including a plurality of candidate recommendation keywords is displayed. Claim 16 In paragraph 15, a method of operation of a display device in which each of the above-mentioned plurality of recommended candidate keywords is a keyword whose reliability with the above-mentioned voice command is greater than or equal to a preset value. Claim 17 delete Claim 18 A method of operation of a display device according to claim 11, wherein a setting menu for setting the exposure of the keyboard window is displayed on the display, and whether the keyboard window is exposed is set according to a command entered through the setting menu. Claim 19 A method of operation of a display device according to claim 11, wherein if the speech source providing the voice command is a remote control device, the keyboard window is displayed on the display, and if the speech source providing the voice command is a user, a recommendation keyword window including a plurality of recommendation keywords is displayed on the display. Claim 20 In claim 19, the above-mentioned plurality of recommended keywords are a method of operation of a display device representing voice commands previously spoken by the user.

Images