Device and system

Abstract

Disclosed is a device for providing a notification by combining state information of a home appliance with an analysis result of content on the chat screen of a messenger platform application. According to an embodiment of the present disclosure, the device may comprise: a communication interface; and a processor which acquires text data corresponding to a user's conversation included on the execution screen of the messenger platform application, recognizes the intention of the acquired text data, acquires information about a home appliance suitable for the recognized intention, generates a notification for guiding control of the home appliance on the basis of the acquired information about the home appliance, and transmits the generated notification to an external device via the communication interface.

Description

Devices and systems

[0001] The present disclosure relates to a device that provides a notification by combining the results of an analysis of the content on the chat screen of a messenger platform application with status information of a home appliance.

[0002] Digital TV services using wired or wireless communication networks are becoming commonplace. Digital TV services can provide a variety of services that were not available through existing analog broadcasting services.

[0003] For example, digital TV services such as IPTV (Internet Protocol Television) and smart TV provide interactivity that allows users to actively select the types of programs and viewing time. Based on this interactivity, IPTV and smart TV services can also provide various additional services, such as internet search, home shopping, and online games.

[0004] Traditionally, when users utilized specific services through mobile apps, they accessed desired functions by navigating menus or entering keywords. Recently, however, with the application of artificial intelligence chatbot technology, a method has been introduced that recognizes the user's natural language input and provides corresponding information or services.

[0005] These chatbot services are provided in the form of messenger platforms or independent applications, and are connected to databases and external services through integration with servers.

[0006] However, existing technologies had limitations in providing a seamless user experience due to the limited scope of service integration or low accuracy in interpreting user input intent.

[0007] The purpose of the present disclosure may be to provide a notification by combining the analysis results of the content on the chat screen of a messenger platform application with the status information of a home appliance.

[0008] The purpose of the present disclosure may be to provide personalized notifications based on information from home appliances collected based on content on a chat screen.

[0009] The purpose of the present disclosure may be to link a chatbot, a server, a display device, and an IoT (Internet of Thinfs) device to recognize a situation based on a user's natural language utterance, automatically analyze the status of the relevant IoT device, and provide a notification.

[0010] An apparatus according to one embodiment of the present disclosure may include a communication interface; and a processor that acquires text data corresponding to a user’s conversation included on an execution screen of a messenger platform application, recognizes the intent of the acquired text data, acquires information of a home appliance corresponding to the recognized intent, generates a notification to guide the control of the home appliance based on the acquired information of the home appliance, and transmits the generated notification to an external device through the communication interface.

[0011] An apparatus according to one embodiment of the present disclosure may include: a user terminal that acquires text data corresponding to a user’s conversation included on an execution screen of a messenger platform application; a server that receives the acquired text data from the user terminal, recognizes the intent of the received text data, acquires information of a home appliance corresponding to the recognized intent, generates a notification to guide the control of the home appliance based on the acquired information of the home appliance, and transmits the generated notification to a display device through the communication interface; and a display device that displays the notification received from the server.

[0012] According to various embodiments of the present disclosure, a much more intuitive and user-friendly interface can be provided than conventional button-based or app-based IoT control methods.

[0013] According to various embodiments of the present disclosure, information delivery capabilities can be greatly improved as information guidance and user feedback interfaces are provided through a lifestyle-oriented large-screen output device called a TV.

[0014] According to various embodiments of the present disclosure, since the user's conversation context and the actual state of the home appliance are considered simultaneously, it is possible to provide more accurate and context-appropriate notifications.

[0015] According to various embodiments of the present disclosure, it is possible to provide customized notifications tailored to the conversation situations and lifestyle patterns of individual users, thereby improving the quality of personalized services.

[0016] According to various embodiments of the present disclosure, users can control the device and check its status using only natural language input without complex commands or operations, thereby improving convenience and accessibility.

[0017] FIG. 1 is a block diagram illustrating the configuration of a display device according to one embodiment of the present invention.

[0018] FIG. 2 is a block diagram of a remote control device according to one embodiment of the present invention.

[0019] FIG. 3 shows an example of the actual configuration of a remote control device according to one embodiment of the present invention.

[0020] Figure 4 shows an example of utilizing a remote control device according to an embodiment of the present invention.

[0021] FIG. 5 is a block diagram illustrating the configuration of a user terminal according to an embodiment of the present invention.

[0022] FIG. 6 is a block diagram illustrating the configuration of a server according to one embodiment of the present disclosure.

[0023] FIG. 7 is a drawing for explaining the configuration of a system according to one embodiment of the present disclosure.

[0024] FIG. 8 is a sequence diagram for explaining a method of providing notifications using a messenger platform of a system according to one embodiment of the present disclosure.

[0025] FIG. 9 is a drawing illustrating an example of providing a notification to a display device using a messenger platform according to one embodiment of the present disclosure.

[0026] FIG. 10 is a sequence diagram for explaining a method of providing notifications using a messenger platform of a system according to another embodiment of the present disclosure.

[0027] FIGS. 11a and FIGS. 11b are drawings illustrating an example of providing a notification to a user terminal using a messenger platform according to another embodiment of the present disclosure.

[0028] FIGS. 12a and 12b are drawings illustrating an example of providing a control service for a related home appliance based on a word entered on the execution screen of a messenger platform app according to another embodiment of the present disclosure.

[0029] FIGS. 13a and 13b are drawings illustrating an example of providing a control service for a related home appliance by analyzing the conversation context on the execution screen of a messenger platform app according to another embodiment of the present disclosure.

[0030] Hereinafter, embodiments related to the present invention will be described in more detail with reference to the drawings. The suffixes "module" and "part" for components used in the following description are assigned or used interchangeably solely for the ease of drafting the specification, and do not inherently have distinct meanings or roles.

[0031] A display device according to an embodiment of the present invention is, for example, an intelligent display device that adds computer support functions to a broadcast reception function. While faithful to the broadcast reception function, it also adds internet functions, and can be equipped with a more user-friendly interface such as a handwriting input device, a touch screen, or a spatial remote control. Furthermore, by supporting wired or wireless internet functions, it can be connected to the internet and a computer, and can perform functions such as email, web browsing, banking, or games. A standardized general-purpose OS may be used for these various functions.

[0032] Accordingly, the display device described in the present invention allows various applications to be freely added or removed, for example, on a general-purpose OS kernel, thereby enabling various user-friendly functions to be performed. More specifically, the display device may be, for example, a network TV, HBB TV, smart TV, LED TV, OLED TV, etc., and may also be applicable to smartphones depending on the circumstances.

[0033] FIG. 1 is a block diagram illustrating the configuration of a display device according to one embodiment of the present invention.

[0034] Referring to FIG. 1, the display device (100) may include a video receiver (130), an external device interface (135), a memory (140), a user input interface (150), a controller (170), a wireless communication interface (173), a display (180), a speaker (185), and a power supply circuit (190).

[0035] The video receiver (130) may include a tuner (131), a demodulator (132), and a network interface (133).

[0036] The tuner (131) can tune to a specific broadcast channel according to a channel tuning command. The tuner (131) can receive a broadcast signal for the tuned specific broadcast channel.

[0037] The demodulator (132) can separate the received broadcast signal into a video signal, an audio signal, and a data signal related to the broadcast program, and can restore the separated video signal, audio signal, and data signal into a form that can be output.

[0038] The external device interface (135) can receive an application or a list of applications within an adjacent external device and transmit it to the controller (170) or memory (140).

[0039] The external device interface (135) can provide a connection path between the display device (100) and an external device. The external device interface (135) can receive one or more of video and audio output from an external device connected to the display device (100) wirelessly or via a wired connection, and transmit them to the controller (170). The external device interface (135) may include a plurality of external input terminals. The plurality of external input terminals may include an RGB terminal, one or more HDMI (High Definition Multimedia Interface) terminals, and a component terminal.

[0040] The video signal of an external device input through the external device interface (135) can be output through the display (180). The audio signal of an external device input through the external device interface (135) can be output through the speaker (185).

[0041] The external device that can be connected to the external device interface (135) may be any one of a set-top box, Blu-ray player, DVD player, game console, soundbar, smartphone, PC, USB memory, or home theater, but this is merely an example.

[0042] The network interface (133) may provide an interface for connecting the display device (100) to a wired / wireless network including the Internet network. The network interface (133) may transmit or receive data to or from other users or other electronic devices through the connected network or another network linked to the connected network.

[0043] In addition, some of the content data stored in the display device (100) can be transmitted to another user or other electronic device selected among other users or other electronic devices that are previously registered in the display device (100).

[0044] The network interface (133) can access a specific web page through a connected network or another network linked to the connected network. That is, it can access a specific web page through a network and transmit or receive data with the corresponding server.

[0045] Additionally, the network interface (133) can receive content or data provided by a content provider or network operator. That is, the network interface (133) can receive content such as movies, advertisements, games, VOD, broadcast signals, and related information provided by a content provider or network provider through a network.

[0046] Additionally, the network interface (133) can receive firmware update information and update files provided by the network operator, and can transmit data to the internet or a content provider or network operator.

[0047] The network interface (133) can select and receive a desired application among the applications that are open to the public through the network.

[0048] The memory (140) can store programs for each signal processing and control within the controller (170), and can store signal-processed video, audio, or data signals.

[0049] Additionally, the memory (140) may perform the function of temporarily storing video, audio, or data signals input from an external device interface (135) or a network interface (133), and may also store information regarding a predetermined image through a channel memory function.

[0050] The memory (140) can store an application or a list of applications input from an external device interface (135) or a network interface (133).

[0051] The display device (100) can play content files (video files, still image files, music files, document files, application files, etc.) stored in memory (140) and provide them to the user.

[0052] The user input interface (150) can transmit a signal input by the user to the controller (170) or transmit a signal from the controller (170) to the user. For example, the user input interface (150) can receive and process control signals such as power on / off, channel selection, and screen setting from the remote control device (200) according to various communication methods such as Bluetooth, Ultra Wideband (WB), ZigBee, Radio Frequency (RF) communication, or Infrared (IR) communication, or process to transmit control signals from the controller (170) to the remote control device (200).

[0053] Additionally, the user input interface (150) can transmit control signals input from local keys (not shown), such as power key, channel key, volume key, and setting value, to the controller (170).

[0054] The image signal processed by the controller (170) can be input to the display (180) and displayed as an image corresponding to the image signal. Additionally, the image signal processed by the controller (170) can be input to an external output device through the external device interface (135).

[0055] The voice signal processed by the controller (170) can be output as audio to the speaker (185). Additionally, the voice signal processed by the controller (170) can be input to an external output device through the external device interface (135).

[0056] In addition, the controller (170) can control the overall operation within the display device (100).

[0057] Additionally, the controller (170) can control the display device (100) by means of user commands or internal programs entered through the user input interface (150), and can connect to a network to allow the user to download an application or a list of applications desired by the user into the display device (100).

[0058] The controller (170) enables the processed video or audio signal, such as channel information selected by the user, to be output through the display (180) or speaker (185).

[0059] Additionally, the controller (170) enables a video signal or audio signal from an external device, such as a camera or camcorder, input through the external device interface (135) according to an external device video playback command received through the user input interface (150), to be output through the display (180) or speaker (185).

[0060] Meanwhile, the controller (170) can control the display (180) to display video, for example, broadcast video input through the tuner (131), external input video input through the external device interface (135), video input through the network interface, or video stored in the memory (140) can be controlled to be displayed on the display (180). In this case, the video displayed on the display (180) may be a still image or a video, and may be a 2D image or a 3D image.

[0061] Additionally, the controller (170) can control the playback of content stored in the display device (100), received broadcast content, or external input content input from the outside, and the content may be in various forms such as broadcast video, external input video, audio file, still image, connected web screen, and document file.

[0062] The wireless communication interface (173) can communicate with an external device via wired or wireless communication. The wireless communication interface (173) can perform short-range communication with an external device. To this end, the wireless communication interface (173) can support short-range communication by using at least one of Bluetooth™, RFID (Radio Frequency Identification), Infrared Data Association (IrDA), Ultra Wideband (UWB), ZigBee, Near Field Communication (NFC), Wireless-Fidelity (Wi-Fi), Wi-Fi Direct, and Wireless Universal Serial Bus (Wireless USB) technologies. Such wireless communication interface (173) can support wireless communication between the display device (100) and a wireless communication system, between the display device (100) and another display device (100), or between the display device (100) and a network where the display device (100, or an external server) is located, through wireless area networks. Short-range wireless communication networks can be Wireless Personal Area Networks.

[0063] Here, another display device (100) may be a user terminal such as a wearable device (e.g., a smartwatch, smart glass, HMD (head mounted display)), a smartphone, etc., capable of exchanging data with (or interacting with) the display device (100) according to the present invention. The wireless communication interface (173) may detect (or recognize) a wearable device capable of communicating around the display device (100).

[0064] Furthermore, if the detected wearable device is a device authenticated to communicate with the display device (100) according to the present invention, the controller (170) can transmit at least a portion of the data processed in the display device (100) to the wearable device through the wireless communication interface (173). Accordingly, the user of the wearable device can use the data processed in the display device (100) through the wearable device.

[0065] The display (180) can generate a driving signal by converting the video signal, data signal, OSD signal processed by the controller (170) or the video signal, data signal, etc. received from the external device interface (135) into R, G, and B signals, respectively.

[0066] Meanwhile, since the display device (100) illustrated in FIG. 1 is merely an embodiment of the present invention, some of the illustrated components may be integrated, added, or omitted depending on the specifications of the actual implemented display device (100).

[0067] That is, as needed, two or more components may be combined into a single component, or a single component may be subdivided into two or more components. In addition, the functions performed in each block are intended to explain embodiments of the present invention, and the specific operations or devices thereof do not limit the scope of the present invention.

[0068] According to another embodiment of the present invention, the display device (100) may receive and play video through a network interface (133) or an external device interface (135) without having a tuner (131) and a demodulator (132), unlike as shown in FIG. 1.

[0069] For example, the display device (100) may be implemented by separating it into a video processing device, such as a set-top box, for receiving broadcast signals or content according to various network services, and a content playback device for playing content input from the video processing device.

[0070] In this case, the operation method of the display device according to the embodiment of the present invention described below may be performed not only by the display device (100) described with reference to FIG. 1, but also by any one of the image processing device such as the separated set-top box, or a content playback device having a display (180) and a speaker (185).

[0071] Next, with reference to FIGS. 2 and 3, a remote control device according to an embodiment of the present invention will be described.

[0072] FIG. 2 is a block diagram of a remote control device according to one embodiment of the present invention, and FIG. 3 shows an example of the actual configuration of a remote control device (200) according to one embodiment of the present invention.

[0073] First, referring to FIG. 2, the remote control device (200) may include a fingerprint reader (210), a wireless communication circuit (220), a user input interface (230), a sensor (240), an output interface (250), a power supply circuit (260), a memory (270), a controller (280), and a microphone (290).

[0074] Referring to FIG. 2, the wireless communication circuit (220) transmits and receives signals with any one of the display devices according to the embodiments of the present invention described above.

[0075] The remote control device (200) is equipped with an RF circuit (221) capable of transmitting and receiving signals to and from the display device (100) according to RF communication standards, and may be equipped with an IR circuit (223) capable of transmitting and receiving signals to and from the display device (100) according to IR communication standards. Additionally, the remote control device (200) may be equipped with a Bluetooth circuit (225) capable of transmitting and receiving signals to and from the display device (100) according to Bluetooth communication standards. Furthermore, the remote control device (200) may be equipped with an NFC circuit (227) capable of transmitting and receiving signals to and from the display device (100) according to NFC (Near Field Communication) communication standards, and may be equipped with a WLAN circuit (229) capable of transmitting and receiving signals to and from the display device (100) according to WLAN (Wireless LAN) communication standards.

[0076] Additionally, the remote control device (200) transmits a signal containing information regarding the movement of the remote control device (200), etc., to the display device (100) through a wireless communication circuit (220).

[0077] Meanwhile, the remote control device (200) can receive a signal transmitted by the display device (100) through the RF circuit (221), and, if necessary, can transmit commands regarding power on / off, channel change, volume change, etc. to the display device (100) through the IR circuit (223).

[0078] The user input interface (230) may be composed of a keypad, buttons, a touchpad, or a touch screen. The user can input commands related to the display device (100) to the remote control device (200) by operating the user input interface (230). If the user input interface (230) is equipped with a hard key button, the user can input commands related to the display device (100) to the remote control device (200) through a push operation of the hard key button. This will be explained with reference to FIG. 3.

[0079] Referring to FIG. 3, the remote control device (200) may include a plurality of buttons. The plurality of buttons may include a fingerprint recognition button (212), a power button (231), a home button (232), a live button (233), an external input button (234), a volume control button (235), a voice recognition button (236), a channel change button (237), a confirmation button (238), and a back button (239).

[0080] The fingerprint recognition button (212) may be a button for recognizing a user's fingerprint. In one embodiment, the fingerprint recognition button (212) may be capable of a push operation and may receive a push operation and a fingerprint recognition operation.

[0081] The power button (231) may be a button for turning the power of the display device (100) on / off.

[0082] The home button (232) may be a button for moving to the home screen of the display device (100).

[0083] The live button (233) may be a button for displaying a live broadcast program.

[0084] The external input button (234) may be a button for receiving an external input connected to the display device (100).

[0085] The volume control button (235) may be a button for adjusting the volume output by the display device (100).

[0086] The voice recognition button (236) may be a button for receiving the user's voice and recognizing the received voice.

[0087] The channel change button (237) may be a button for receiving a broadcast signal of a specific broadcast channel.

[0088] The confirmation button (238) may be a button for selecting a specific function, and the back button (239) may be a button for returning to the previous screen.

[0089] Figure 2 is explained again.

[0090] If the user input interface (230) is equipped with a touchscreen, the user can input commands related to the display device (100) to the remote control device (200) by touching the soft keys on the touchscreen. Additionally, the user input interface (230) may be equipped with various types of input means that the user can operate, such as scroll keys or jog keys, and this embodiment does not limit the scope of the present invention.

[0091] The sensor (240) may be equipped with a gyroscope sensor (241) or an accelerometer sensor (243), and the gyroscope sensor (241) may sense information regarding the movement of the remote control device (200).

[0092] For example, the gyroscope sensor (241) can sense information regarding the operation of the remote control device (200) based on the x, y, and z axes, and the accelerometer sensor (243) can sense information regarding the movement speed of the remote control device (200). Meanwhile, the remote control device (200) may further be equipped with a distance measuring sensor to sense the distance to the display (180) of the display device (100).

[0093] The output interface (250) can output a video or audio signal corresponding to the operation of the user input interface (230) or a signal transmitted from the display device (100).

[0094] The user can recognize whether the output interface (250) is operated by the user input interface (230) or whether the display device (100) is controlled.

[0095] For example, the output interface (250) may be equipped with an LED (251) that lights up when the user input interface (230) is operated or when a signal is transmitted or received with the display device (100) through the wireless communication circuit (220), a vibrator (253) that generates vibration, a speaker (255) that outputs sound, or a display (257) that outputs video.

[0096] Additionally, the power supply circuit (260) supplies power to the remote control device (200), and can reduce power waste by stopping the power supply when the remote control device (200) does not move for a predetermined period of time.

[0097] The power supply circuit (260) can resume power supply when a predetermined key provided in the remote control device (200) is operated.

[0098] The memory (270) can store various types of programs, application data, etc. required for the control or operation of the remote control device (200).

[0099] When the remote control device (200) transmits and receives signals wirelessly through the RF circuit (221) and the display device (100), the remote control device (200) and the display device (100) transmit and receive signals through a predetermined frequency band.

[0100] The controller (280) of the remote control device (200) can store and refer to information regarding frequency bands, etc., that can wirelessly transmit and receive signals with the display device (100) paired with the remote control device (200) in the memory (270).

[0101] The controller (280) controls all matters related to the control of the remote control device (200). The controller (280) can transmit a signal corresponding to a predetermined key operation of the user input interface (230) or a signal corresponding to the movement of the remote control device (200) sensed by the sensor (240) to the display device (100) through the wireless communication circuit (220).

[0102] In addition, the microphone (290) of the remote control device (200) can acquire voice.

[0103] The microphone (290) may be provided in multiple numbers.

[0104] Next, Figure 4 is explained.

[0105] Figure 4 shows an example of utilizing a remote control device according to an embodiment of the present invention.

[0106] FIG. 4(a) illustrates a pointer (205) corresponding to a remote control device (200) being displayed on a display (180).

[0107] The user can move or rotate the remote control device (200) up and down, left and right. The pointer (205) displayed on the display (180) of the display device (100) corresponds to the movement of the remote control device (200). Since the pointer (205) of the remote control device (200) moves and is displayed according to the movement in 3D space as shown in the drawing, it can be named a spatial remote control.

[0108] FIG. 4(b) illustrates that when the user moves the remote control device (200) to the left, the pointer (205) displayed on the display (180) of the display device (100) also moves to the left in response.

[0109] Information regarding the movement of the remote control device (200) detected through the sensor of the remote control device (200) is transmitted to the display device (100). The display device (100) can calculate the coordinates of the pointer (205) from the information regarding the movement of the remote control device (200). The display device (100) can display the pointer (205) to correspond to the calculated coordinates.

[0110] FIG. 4(c) illustrates a case where, while pressing a specific button within the remote control device (200), the user moves the remote control device (200) away from the display (180). By doing so, the selected area within the display (180) corresponding to the pointer (205) can be zoomed in and enlarged.

[0111] Conversely, when the user moves the remote control device (200) closer to the display (180), the selected area within the display (180) corresponding to the pointer (205) can be zoomed out and reduced in size.

[0112] Meanwhile, when the remote control device (200) moves away from the display (180), the selected area may be zoomed out, and when the remote control device (200) moves closer to the display (180), the selected area may be zoomed in.

[0113] Additionally, when a specific button within the remote control device (200) is pressed, recognition of up-down and left-right movement may be excluded. That is, when the remote control device (200) moves away from or closer to the display (180), up-down, left-right movement is not recognized, and only forward-backward movement is recognized. When the specific button within the remote control device (200) is not pressed, only the pointer (205) moves according to the up-down, left-right movement of the remote control device (200).

[0114] Meanwhile, the movement speed or direction of movement of the pointer (205) can correspond to the movement speed or direction of movement of the remote control device (200).

[0115] Meanwhile, the pointer in this specification refers to an object displayed on the display (180) in response to the operation of the remote control device (200). Accordingly, the pointer (205) can be an object of various shapes other than the arrow shape shown in the drawing. For example, it may be a concept including a point, a cursor, a prompt, a thick outline, etc. Furthermore, the pointer (205) can be displayed corresponding to either a point on the horizontal axis or a vertical axis on the display (180), and it is also possible to display it corresponding to multiple points, such as a line or a surface.

[0116] FIG. 5 is a block diagram illustrating the configuration of a user terminal according to an embodiment of the present invention.

[0117] The user terminal (500) can be implemented as a fixed device or a mobile device, such as a projector, mobile phone, smartphone, desktop computer, laptop, digital broadcasting terminal, PDA (personal digital assistants), PMP (portable multimedia player), navigation, tablet PC, wearable device, set-top box (STB), DMB receiver, radio, desktop computer, etc.

[0118] Referring to FIG. 5, the user terminal (500) may include a communication circuit (510), an input interface (520), a memory (530), an output interface (540), and a processor (590).

[0119] The communication circuit (510) can transmit and receive data with external devices such as other user terminals or servers using wired and wireless communication technology.

[0120] The communication circuit (510) can perform communication using any one of the following communication standards: GSM (Global System for Mobile communication), CDMA (Code Division Multi Access), LTE (Long Term Evolution), 5G, WLAN (Wireless LAN), Wi-Fi (Wireless-Fidelity), Bluetooth (Bluetooth), RFID (Radio Frequency Identification), Infrared Data Association (IrDA), ZigBee, and NFC (Near Field Communication).

[0121] The input interface (520) may include a camera (521) for inputting a video signal, a microphone (522) for receiving an audio signal, a user input interface for receiving information from a user, etc.

[0122] Here, the camera (521) or microphone (522) may be treated as a sensor, and the signal obtained from the camera (521) or microphone (522) may be referred to as sensing data or sensor information.

[0123] The memory (530) can store various software and data related to the operation of the user terminal (500).

[0124] The output interface (540) may include a display (541) that displays a video signal received from the outside and a speaker (542) that outputs an audio signal.

[0125] The processor (590) can control the overall operation of the user terminal (500).

[0126] When the processor (590) requires the connection of an external device to perform the operation of the user terminal (500), it can generate a control signal to control the external device and transmit the generated control signal to the external device.

[0127] The processor (590) can control at least some of the components of the user terminal (500) to run an application stored in memory (530).

[0128] The processor (590) can operate two or more of the components included in the user terminal (500) in combination with each other to run the application program.

[0129] FIG. 6 is a block diagram illustrating the configuration of a server according to one embodiment of the present disclosure.

[0130] Referring to FIG. 6, the server (600) may include a communication interface (610), a database (630), and a processor (690).

[0131] The server (600) may be referred to as the server device (600) or the device (600).

[0132] The communication interface (610) can communicate with any one of the display device (100), home appliance (700), and user terminal (500) through a wireless communication standard.

[0133] Any one of the wireless communication standards, such as Wi-Fi, 4G LTE, or 5G, may be used, and for this purpose, the communication interface (610) may be equipped with one or more communication circuits.

[0134] The database (630) may store information on each of the multiple home appliances registered on the server (600).

[0135] The processor (690) can control the overall operation of the server (600).

[0136] The processor (690) can control multiple registered home appliances through the display device (100) or user terminal (500).

[0137] FIG. 7 is a drawing for explaining the configuration of a system according to one embodiment of the present disclosure.

[0138] Referring to FIG. 7, the system (70) may include a user terminal (500), a server (600), a display device (100), and a home appliance (700).

[0139] The user terminal (500) can display the execution screen of a messenger platform application (hereinafter referred to as “app”).

[0140] The user terminal (500) can obtain an input phrase through user input on an execution screen and transmit text data corresponding to the obtained input phrase to the server (600) through a communication circuit (510).

[0141] The server (600) can obtain the analysis results of the text data received from the user terminal (500).

[0142] The server (600) can obtain information about home appliances that match the user's intention based on the acquired analysis results.

[0143] The server (600) can generate a notification based on the information of the acquired home appliance and can transmit the generated notification to the display device (100) through the communication interface (610).

[0144] The display device (100) can display a notification received from the server (600) on the display (180).

[0145] The display device (100) can transmit control commands based on user input received through the displayed notification to the server (600) via the network interface (133).

[0146] The server (600) can transmit control commands received from the display device (100) to the home appliance (700) through the communication interface (610).

[0147] The home appliance (700) can perform operations according to control commands received from the server (600).

[0148] The server (600) can generate a notification based on the information of the acquired home appliance and can transmit the generated notification to the user terminal (500) through the communication interface (610).

[0149] The user terminal (500) can display the notification received from the server (600) on the display (541).

[0150] The server (600) can transmit control commands received from the user terminal (500) to the home appliance (700) through the communication interface (610).

[0151] The home appliance (700) can perform operations according to control commands received from the server (600).

[0152] From the perspective of the server (600), the user terminal (500), the display device (100), and the home appliance (700) can each be referred to as external devices.

[0153] FIG. 8 is a sequence diagram for explaining a method of providing notifications using a messenger platform of a system according to one embodiment of the present disclosure.

[0154] In particular, FIG. 8 is an embodiment in which a notification for controlling the status and function of a home appliance corresponding to a user’s conversation entered through a messenger platform is provided to a display device (100).

[0155] Referring to FIG. 8, the processor (590) of the user terminal (500) can display the execution screen of a messenger platform app (S801).

[0156] A messenger platform app may be a network-based app that enables the exchange of messages between user terminals (500). A messenger platform app may provide a chatbot function. The chatbot function may be a function that analyzes user input, such as text or voice, to automatically generate a response or performs an operation in conjunction with an external device or service.

[0157] The execution screen of a messenger platform app can be referred to as a chatbot screen. The execution screen of a messenger platform app can be a conversation screen for conversing with another person.

[0158] The processor (590) of the user terminal (500) can obtain an input phrase through user input on the execution screen and transmit text data corresponding to the obtained input phrase to the server (600) through the communication circuit (510) (S803).

[0159] The processor (590) can obtain either text corresponding to the user's voice command or text entered through a virtual keyboard as an input phrase on the execution screen of the messenger platform app. The processor (590) can convert the voice command into text and obtain the converted text as an input phrase. The processor (590) can obtain text typed by the user through a virtual keyboard window displayed on the execution screen as an input phrase.

[0160] The processor (590) can transmit text data corresponding to the text corresponding to the input phrase to the server (600) through the communication circuit (510). The communication circuit (510) may include a circuit having a communication standard that supports internet communication. For example, the communication circuit (510) may be a circuit of Wi-Fi communication standard.

[0161] The processor (690) of the server (600) can obtain the analysis result of the text data received from the user terminal (500) (S805).

[0162] The processor (690) can obtain an analysis result by analyzing text data using a natural language processing engine. The analysis result may include the user's intent regarding the input phrase.

[0163] For example, the analysis results may include one or more of the intentions to control a home appliance or a specific home appliance.

[0164] The processor (690) of the server (600) can obtain information about home appliances that match the user's intention based on the acquired analysis results (S807).

[0165] Information about a home appliance may include one or more of the identification information of the home appliance, status information of the home appliance, or function information of the home appliance.

[0166] The identification information of a home appliance may include at least one of the name of the home appliance, the model name of the home appliance, or the identifier of the home appliance.

[0167] The status information of a home appliance may include one or more of the power status or operating status of the home appliance.

[0168] Function information of a home appliance may include at least one of the multiple functions possessed by the home appliance.

[0169] The processor (690) can use the analysis results to extract a home appliance that matches the user's intention among a plurality of connected home appliances. For example, if the analysis results include an intention to perform laundry, the processor (690) can extract a home appliance that matches the user's intention as a washing machine. The processor (690) can obtain the status information and function information of the extracted washing machine as information of the home appliance.

[0170] The processor (690) of the server (600) can generate a notification based on the information of the acquired home appliance (S809), and can transmit the generated notification to the display device (100) through the communication interface (610) (S811).

[0171] The notification may be intended to guide the control of home appliances.

[0172] The processor (690) can generate a notification based on the status information of the home appliance and the function information of the home appliance. For example, the processor (690) can generate a notification including the power status of the home appliance and the automatic scheduling setting function of the home appliance.

[0173] The controller (170) of the display device (100) can display a notification received from the server (600) on the display (180) (S813).

[0174] The controller (170) of the display device (100) can transmit control commands based on user input received through the displayed notification to the server (600) via the network interface (133) (S815).

[0175] The control command may be a command to control the operation of the home appliance (700).

[0176] The control command may be a command for controlling at least one of the power status of the home appliance (700) or the functions of the home appliance.

[0177] The processor (690) of the server (600) can transmit control commands received from the display device (100) to the home appliance (700) through the communication interface (610) (S817).

[0178] The home appliance (700) can perform operations according to control commands received from the server (600) (S819).

[0179] Hereinafter, an embodiment of FIG. 8 will be explained with reference to FIG. 9.

[0180] FIG. 9 is a drawing illustrating an example of providing a notification to a display device using a messenger platform according to one embodiment of the present disclosure.

[0181] Referring to FIG. 9, a washing machine (700-1) is shown as an example of a home appliance (700).

[0182] The user terminal (500) can display the execution screen (900) of a messenger platform app on the display (541). The user terminal (500) can display an input phrase (901) based on user input on the execution screen (900).

[0183] The user terminal (500) can transmit text data corresponding to the input phrase (901) to the server (600). The input phrase (901) is It may be a conversation phrase that reflects the user's intention to do laundry.

[0184] In one embodiment, a linkage server (not shown) linked with a messenger platform app may be separately provided. The linkage server may receive text data received from a user terminal (500) and transmit the received text data to a server (600). As another example, the linkage server may obtain an analysis result for the received text data and transmit the obtained analysis result to the server (600).

[0185] The server (600) can obtain an analysis result including the user's intent from text data received from the user terminal (500) using a natural language processing engine.

[0186] The server (600) can extract keywords from text data corresponding to the input phrase (901) and can identify home appliances that match the extracted keywords. The server (600) can extract keywords from the text data corresponding to the input phrase (901). <laundry>It can be obtained as, <laundry>You can obtain a home appliance related to this as a washing machine (700-1).

[0187] The server (600) can obtain status information of the washing machine (700-1), including the power status and operation status of the washing machine (700-1), and information of the washing machine (700-1), including one or more functions possessed by the washing machine (700-1), based on extracted keywords. For example, the information of the washing machine (700-1) may include a power-off state in which the power of the washing machine (700-1) is turned off and an automatic reservation function.

[0188] The server (600) can generate a notification including a power-off state and an automatic reservation function, and can transmit the generated notification to the display device (100). The display device (100) can display the received notification (910) on the display (180). The notification (910) may include text guiding the power state of the washing machine (700-1) and the automatic reservation function of the washing machine (700-1) that best matches the intent of the input phrase (901) entered on the execution screen (900) of the messenger platform app of the user terminal (500).

[0189] The notification (910) may include a consent button (911) for setting the automatic reservation function of the washing machine (700-1). The display device (100) may send a control command to the server (600) to set the automatic reservation function of the washing machine (700-1) according to the command to select the consent button (911).

[0190] The server (600) can transmit a control command received from the display device (100) to the washing machine (700-1). The washing machine (700-1) can perform a setting operation to start washing at a specific time according to the received control command.

[0191] In this way, according to an embodiment of the present disclosure, information guidance and user feedback interfaces are provided through a living-oriented large-screen output device called a TV, thereby greatly improving the ability to convey information.

[0192] According to an embodiment of the present disclosure, a user can control the device and check its status using only natural language input without complex commands or operations, thereby improving convenience and accessibility.

[0193] FIG. 10 is a sequence diagram for explaining a method of providing notifications using a messenger platform of a system according to another embodiment of the present disclosure.

[0194] In particular, FIG. 10 is an embodiment that provides a notification to a user terminal (500) for controlling the status and function of a home appliance in accordance with a user's command entered through a messenger platform.

[0195] In the following, since each of steps S1001 to S1009 is identical to each of steps S801 to S809 of FIG. 8, a detailed description is omitted.

[0196] Referring to FIG. 10, the processor (590) of the user terminal (500) can display the execution screen of a messenger platform app (S1001).

[0197] The processor (590) of the user terminal (500) can obtain an input phrase through user input on the execution screen and transmit text data corresponding to the obtained input phrase to the server (600) through the communication circuit (510) (S1003).

[0198] The processor (690) of the server (600) can obtain the analysis result of the text data received from the user terminal (500) (S1005).

[0199] The processor (690) of the server (600) can obtain information about home appliances that match the user's intention based on the acquired analysis results (S1007).

[0200] The processor (690) of the server (600) can generate a notification based on the information of the acquired home appliance (S1009), and can transmit the generated notification to the user terminal (500) through the communication interface (610) (S1011).

[0201] The processor (590) of the user terminal (500) can display the notification received from the server (600) on the display (541) (S1013).

[0202] The processor (590) can display a notification on the execution screen of the messenger platform app in response to an input phrase. That is, the notification may be a conversational response following the input phrase.

[0203] The processor (590) of the user terminal (500) can transmit control commands based on user input received through a displayed notification to the server (600) via the network interface (133) (S1015).

[0204] The control command may be a command for controlling at least one of the power status of the home appliance (700) or the functions of the home appliance.

[0205] The processor (690) of the server (600) can transmit control commands received from the user terminal (500) to the home appliance (700) through the communication interface (610) (S1017).

[0206] The home appliance (700) can perform operations according to control commands received from the server (600) (S1019).

[0207] Hereinafter, an embodiment of FIG. 10 will be described with reference to FIG. 11a and FIG. 11b.

[0208] FIGS. 11a and FIGS. 11b are drawings illustrating an example of providing a notification to a user terminal using a messenger platform according to another embodiment of the present disclosure.

[0209] Referring to FIG. 11a, a user terminal (500) can display a first execution screen (1100) of a messenger platform app. The user terminal (500) can display a first input phrase (1111) corresponding to a received user input on the first execution screen (1110). The first input phrase (1111) is<weather is good> It may be a phrase reflecting the intention to control home appliances related to the weather.

[0210] A user terminal (500) can transmit first text data corresponding to a first input phrase (1111) to a server (600), and the server (600) can obtain a first analysis result of the first text data received from the user terminal (500) through a natural language processing engine.

[0211] The server (600) can extract keywords from the first text data, and the keywords of the first input phrase (1111)<weather, good> It can be extracted. The server (600) can obtain the control intention of the home appliance related to clear weather as the first analysis result.

[0212] The server (600) may acquire a washing machine related to clear weather among a plurality of home appliances equipped in the home as a control intent target, and may acquire one or more of the washing machine's status information or the washing machine's function information. The database (630) of the server (600) may store information for each of the plurality of home appliances registered in the server (600).

[0213] The server (600) can generate a first notification to guide the operation control of the washing machine based on the acquired washing machine information, and can transmit the generated first notification to the user terminal (500).

[0214] As illustrated in FIG. 11a, the user terminal (500) can display the first notification (1112) received from the server (600) on the first execution screen (1110) of the messenger platform app.

[0215] The first notification (1112) may be a conversational phrase that responds to the first input phrase (1111).

[0216] The first notification (1112) may include a washing machine image (1112a) representing a washing machine corresponding to the first input phrase (1111), guide text (1112b) indicating that the washing machine is controllable, a start item (1112c) for initiating the operation of the washing machine, and a status check item (1112d) for determining the status of the washing machine. The first notification (1112) may further include a function item for controlling the main functions of the washing machine.

[0217] When a start item (1112c) is selected, the user terminal (500) can transmit a control command to the server (600) to start a washing operation of the washing machine. The server (600) can transmit the control command received from the user terminal (500) to the washing machine, and the washing machine can start a washing operation according to the received control command.

[0218] When a status check item (1112d) is selected, the user terminal (500) can display the washing machine status information on the first execution screen (1110). The washing machine status information may include power status information indicating whether the washing machine's power is on or off, and operation status information indicating the washing machine's operation status.

[0219] As such, according to an embodiment of the present disclosure, a user can control the device and check its status using only natural language input without complex commands or operations, thereby improving convenience and accessibility.

[0220] Referring to FIG. 11b, the user terminal (500) can display a second execution screen (1120) of a messenger platform app. The user terminal (500) can display a second input phrase (1121) corresponding to the received user input on the second execution screen (1120). The second input phrase (1121) is<Fine dust is severe> This may be a phrase reflecting the intention to control home appliances related to fine dust.

[0221] The user terminal (500) can transmit second text data corresponding to the second input phrase (1121) to the server (600), and the server (600) can obtain the second analysis result of the second text data received from the user terminal (500) through a natural language processing engine.

[0222] The server (600) can extract keywords from the second text data, and the keywords of the second input phrase (1121)<Fine dust, severe> It can be extracted. The server (600) can obtain the control intention of the fine dust-related home appliance as a second analysis result.

[0223] The server (600) can acquire a fine dust-related air purifier among a plurality of household appliances equipped in the home as a control intent target, and can acquire one or more of the status information of the air purifier or the function information of the air purifier.

[0224] The server (600) can generate a second notification to guide the operation control of the air purifier based on the acquired information of the air purifier, and can transmit the generated second notification to the user terminal (500).

[0225] As illustrated in FIG. 11b, the user terminal (500) can display the second notification (1122) received from the server (600) on the second execution screen (1120) of the messenger platform app.

[0226] The second notification (1122) may be a conversational phrase that responds to the second input phrase (1121).

[0227] The second notification (1122) may include an air purifier image (1122a) representing an air purifier corresponding to the second input phrase (1121), guide text (1122b) indicating that the air purifier can be controlled, a start item (1122c) for starting the operation of the air purifier, and a status check item (1122d) for determining the status of the air purifier.

[0228] When the initiation item (1122c) is selected, the user terminal (500) can transmit a control command to the server (600) to initiate the air circulation operation of the air purifier. The server (600) can transmit the control command received from the user terminal (500) to the air purifier, and the air purifier can initiate the air circulation operation according to the received control command.

[0229] When a status check item (1122d) is selected, the user terminal (500) can display status information of the air purifier on the second execution screen (1120). The status information of the air purifier may include power status information indicating whether the power of the air purifier is on or off, and operation status information indicating the operation status of the air purifier.

[0230] As such, according to an embodiment of the present disclosure, a user can control the device and check its status using only natural language input without complex commands or operations, thereby improving convenience and accessibility.

[0231] Meanwhile, the messenger platform app may also be installed on the display device (100). In this case, the first execution screen (1110) and the second execution screen (1120) of FIG. 11a may each be a screen displayed on the display (180) of the display device (100). In addition, in this case, the display device (100) may receive notifications (1112, 1121) from the server (600) and display the received notifications (1112, 1121) on the display (180).

[0232] FIGS. 12a and 12b are drawings illustrating an example of providing a control service for a related home appliance based on a word entered on the execution screen of a messenger platform app according to another embodiment of the present disclosure.

[0233] The execution screen (1210, 1220) of the messenger platform app shown in FIG. 12a and FIG. 12b can be displayed on a user terminal (500) or a display device (100).

[0234] In the following, it is assumed that the execution screen (1210, 1220) of the messenger platform app is displayed on a user terminal (500).

[0235] In FIGS. 12a and 12b, the user can register one or more home appliances connected to the display device (100) or the display device (100) through a messaging platform app. When registering home appliances, the user can register the name of the home appliance, the type of home appliance, or the main function of the home appliance as a word for providing effects. The word for providing effects may be a word that provides an animation effect when the name, type, or main function is entered on the execution screen.

[0236] Words for providing effects can be stored in the database (630) of the server (600).

[0237] A user terminal (500) or server (600) may store one or more words for providing effects and animation effects matching those words in memory (530) or a database (630).

[0238] For example, if the server (600) is a washing machine, the type of home appliance is a washing machine.<washing machine> and<washing machine> You can save an animation effect where a washing machine matching it appears.

[0239] Referring to FIG. 12a, a third execution screen (1210) of a messenger platform app is shown. The third execution screen (1210) may include a third input phrase (1211) displayed according to user input. The third input phrase (1211) may be a phrase indicating the type of home appliance.

[0240] In one embodiment, when the user terminal (500) recognizes the third input phrase (1211), the data stored in the memory (530)<washing machine> An animation effect (1212) in which a washing machine matching the one appears can be displayed on the third execution screen (1210).

[0241] The user terminal (500) may display a pop-up notification (1112) on the third execution screen (1210) when either the third input phrase (1211) or the animation effect (1212) is selected. The pop-up notification (1112) may be the first notification (1112) for guiding the operation control of the washing machine described in FIG. 11a.

[0242] In another embodiment, the user terminal (500) can transmit text data corresponding to the third input phrase (1211) to the server (600), and the server (600) can obtain the control intention of the washing machine as an analysis result based on the text data corresponding to the third input phrase (1211). The server (600) can generate a popup notification (1112) based on the obtained control intention. The server (600) can transmit the generated popup notification (1112) to the user terminal (500), and the user terminal (500) can display the popup notification (1112) received from the server (600) on the third execution screen (1210).

[0243] Referring to FIG. 12b, a fourth execution screen (1220) of a messenger platform app is shown. The fourth execution screen (1220) may include a third input phrase (1211) displayed according to user input. The third input phrase (1211) may be a phrase indicating the type of home appliance.

[0244] In one embodiment, when the user terminal (500) recognizes the third input phrase (1211), the data stored in the memory (530)<washing machine> An animation effect (1212) in which a washing machine matching the one appears can be displayed on the fourth execution screen (1220).

[0245] When either the third input phrase (1211) or the animation effect (1212) is selected, the user terminal (500) may display a link (1221) on the fourth execution screen (1221) that provides access to the screen of a chatbot application for controlling the connection and operation of the washing machine.

[0246] FIGS. 13a and 13b are drawings illustrating an example of providing a control service for a related home appliance by analyzing the conversation context on the execution screen of a messenger platform app according to another embodiment of the present disclosure.

[0247] In FIG. 13a and FIG. 13b, the execution screen (1310, 1320) of the messenger platform app can be displayed on a user terminal (500) or a display device (100).

[0248] In the following, it is assumed that the execution screen (1310, 1320) of the messenger platform app is displayed on a user terminal (500).

[0249] A user terminal (500) can transmit text data corresponding to input phrases (1311, 1312) to a server (600), and the server (600) can obtain a control intention of a home appliance that matches the conversation context based on the text data as a result of analyzing the conversation. Based on the text data, the server (600) can obtain a control target device as an air purifier, and can obtain a control intention of the air purifier as a result of analyzing the conversation.

[0250] The server (600) can generate a notification based on the analysis results and can transmit the generated notification to the user terminal (500). The user terminal (500) can display the notification (1313) on the fifth execution screen (1310). The notification (1313) may include one or more of a function item (1313a) indicating the function of the air purifier or a link item (1313b) for accessing a chatbot screen for controlling the air purifier.

[0251] The user can easily control the air purifier through the function item (1313a) and control the air purifier in detail through the link item (1313b).

[0252] Referring to FIG. 13b, the sixth execution screen (1320) may include a conversation input window (1321). The user terminal (500) may transmit text data corresponding to an input phrase (1322) entered on the conversation input window (1321) to the server (600), and the server (600) may obtain the analysis result of the received text data.

[0253] The server (600) can obtain the control intention of the air purifier as an analysis result based on the input phrase (1322) and can send a control guide notification based on the analysis result to the user terminal (500).

[0254] The user terminal (500) can display an air purifier emoticon (1323) representing the air purifier to be controlled on the conversation input window (1321) according to the control guide notification.

[0255] The user terminal (500) can display a function item (1313a) indicating the function of the air purifier or a link item (1313b) for accessing a chatbot screen for controlling the air purifier on the enlarged window provided through the enlarged menu of the 6th execution screen (1320).

[0256] According to one embodiment of the present invention, the above-described method can be implemented as code that a processor can read on a medium on which a program is recorded. Examples of a medium that a processor can read include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc.

[0257] The display device described above is not limited to the configuration and method of the embodiments described above; rather, all or part of each embodiment may be selectively combined to allow for various modifications to be made.< / laundry> < / laundry>

Claims

1. In the device, Communication interface; and A processor comprising: acquiring text data corresponding to a user’s conversation included on the execution screen of a messenger platform application; recognizing the intent of the acquired text data; acquiring information of a home appliance matching the recognized intent; generating a notification to guide the control of the home appliance based on the acquired information of the home appliance; and transmitting the generated notification to an external device through the communication interface. device.

2. In Paragraph 1, The above processor Receiving a control command to control the operation of the home appliance from the external device through the communication interface and transmitting the received control command to the home appliance. device.

3. In Paragraph 1, The above processor Extracting one or more keywords from the text data using a natural language processing engine, and recognizing the intent using the extracted one or more keywords. device.

4. In Paragraph 1, The information of the above home appliance is including one or more of status information of the above-mentioned home appliance or function information regarding the functions possessed by the above-mentioned home appliance. device.

5. In Paragraph 4, The above notification is Includes an item for providing the above-mentioned status information of the above-mentioned home appliance and an item for controlling the above-mentioned function. device.

6. In Paragraph 4, The above notification is Includes a link that provides access to the screen of a chatbot application for controlling the above-mentioned home appliance. device.

7. In Paragraph 1, The above processor Receiving the text data corresponding to the conversation from the external device device.

8. In Paragraph 1, The above external device is Either a TV or a user terminal on which the above messenger platform application is installed device.

9. In the system, A user terminal that acquires text data corresponding to a user's conversation included on the execution screen of a messenger platform application; A server that receives the acquired text data from the user terminal, recognizes the intent of the received text data, acquires information of a home appliance corresponding to the recognized intent, generates a notification to guide the control of the home appliance based on the acquired information of the home appliance, and transmits the generated notification to a display device; and A display device that displays notifications received from the above server System.

10. In Paragraph 9, The above server is Receiving a control command to control the operation of the home appliance from the display device and transmitting the received control command to the home appliance System.

11. In Paragraph 9, The above server is Extracting one or more keywords from the text data using a natural language processing engine, and recognizing the intent using the extracted one or more keywords. System.

12. In Paragraph 9, The information of the above home appliance is including one or more of status information of the above-mentioned home appliance or function information regarding the functions possessed by the above-mentioned home appliance. System.

13. In Paragraph 12, The above notification is Includes an item for providing the above-mentioned status information of the above-mentioned home appliance and an item for controlling the above-mentioned function. System.

14. In Paragraph 12, The above notification is Includes a link that provides access to the screen of a chatbot application for controlling the above-mentioned home appliance. System.

15. In Paragraph 9, The above server is Transmitting the above-generated notification to the user terminal System.

Images