Method and apparatus for providing voice assistant service

Abstract

A method for providing a voice assistant service according to one embodiment of the present disclosure may include: receiving a first voice command from a user; determining a plurality of devices to transmit the first voice command among a plurality of candidate devices; transmitting information related to the first voice command to a plurality of devices; receiving a plurality of service provision messages responding to the first voice command and service provision history information of the plurality of devices from each of the plurality of devices; selecting at least one of the plurality of service provision messages based on the service provision history information; and outputting a response message responding to the first voice command based on the selected result.

Description

Technology Field

[0001] The present disclosure relates to a method and apparatus for providing a voice assistant service, and more specifically, to a method and apparatus for providing a voice assistant service that recommends at least one service among a plurality of services that can be provided by a plurality of devices in response to a voice command received from a user. Background Technology

[0002] With the recent development of electronic devices that perform a variety of complex functions, such as smartphones, voice-recognition capable electronic devices are being released to improve operability.

[0003] Speech recognition technology can provide user-friendly conversational services by being applied to a Conversational User Interface that outputs response messages in response to voice questions entered by the user in everyday language. A Conversational User Interface refers to an intelligent user interface that operates while conversing in the user's language.

[0004] For example, electronic devices such as smart speakers, smartphones, computers, PDAs (personal digital assistants), PMPs (portable multimedia players), smart home appliances, navigation systems, and wearable devices can provide conversation services by connecting to a server or running an application.

[0005] Furthermore, as Artificial Intelligence (AI) technology advances, AI technology has been integrated into speech recognition functions, enabling fast and accurate speech recognition for various utterances. An AI system is a computer system that implements human-level intelligence; unlike existing rule-based smart systems, it is a system in which the machine learns, makes judgments, and becomes smarter on its own. As AI systems improve in recognition rate and become capable of understanding user preferences more accurately with continued use, existing rule-based smart systems are gradually being replaced by deep learning-based AI systems. The problem to be solved

[0006] As the number of devices equipped with voice recognition capabilities to provide conversational services increases, the range and variety of services available to users are also growing. Consequently, there is a difficulty in that users must familiarize themselves with and utter all the numerous commands for each device in order to select and receive the services they desire. means of solving the problem

[0007] In order to solve these problems, according to various embodiments of the present disclosure, a method is proposed in which a voice assistant service providing device connected to a plurality of devices groups the devices according to the capabilities of the devices in response to a user's voice command, selects an optimal service from among the services that the devices can provide, and proposes it to the user.

[0008] A method for providing a voice assistant service according to one embodiment of the present disclosure may include: receiving a first voice command from a user; determining a plurality of devices among a plurality of candidate devices to transmit information related to the first voice command; transmitting information related to the first voice command to the plurality of devices; receiving a plurality of service provision messages responding to the first voice command and service provision history information of the plurality of devices from each of the plurality of devices; selecting at least one of the plurality of service provision messages based on the service provision history information; and outputting a response message responding to the first voice command based on the selected result.

[0009] A voice assistant service providing device according to one embodiment of the present disclosure comprises: a receiving unit for receiving a voice command from a user; a communication unit; a memory for storing one or more commands; and at least one processor for executing a voice assistant program by executing the one or more commands. The at least one processor controls the receiving unit to receive a first voice command from the user, determines a plurality of devices among a plurality of candidate devices to transmit information related to the first voice command, transmits information related to the first voice command to the plurality of devices, controls the communication unit to receive a plurality of service providing messages responding to the first voice command and service providing history information of the plurality of devices from each of the plurality of devices, selects at least one of the plurality of service providing messages based on the service providing history information, and outputs a response message responding to the first voice command based on the selected result. Effects of the invention

[0010] A voice assistant service providing device according to one embodiment suggests an optimal service preferred by the user among various services provided by a plurality of devices, thereby enabling the user to conveniently communicate with and control a large number of devices. Brief explanation of the drawing

[0011] FIG. 1a illustrates a system providing a voice assistant service according to one embodiment of the present disclosure. FIG. 1b illustrates an example of a block diagram of a system providing a voice assistant service according to one embodiment of the present disclosure. FIG. 2 illustrates a signal flow diagram of an operation method of a system providing a voice assistant service according to one embodiment of the present disclosure. FIG. 3 illustrates a signal flow diagram of an operation method of a system providing a voice assistant service according to one embodiment of the present disclosure. FIG. 4 illustrates a flowchart of a method for providing a voice assistant service according to one embodiment of the present disclosure. FIG. 5 illustrates a specific flowchart of a method for providing a voice assistant service according to one embodiment of the present disclosure. FIG. 6 illustrates a flowchart of a method of operation of a device interacting with a user through a voice assistant service providing device according to one embodiment of the present disclosure. FIG. 7 illustrates a specific flowchart of a method of operation of a device interacting with a user through a voice assistant service providing device according to one embodiment of the present disclosure. FIG. 8 illustrates an example of data that a voice assistant service provider learns according to one embodiment of the present disclosure. FIG. 9 illustrates examples of a service provision message, service provision history information, and device information received by a voice assistant service providing device according to one embodiment of the present disclosure. FIG. 10 illustrates a flowchart of a method in which a voice assistant service providing device assigns priority to a plurality of service providing messages according to one embodiment of the present disclosure. FIG. 11 illustrates an example of a specific operation method of a system providing a voice assistant service according to a first embodiment of the present disclosure. FIG. 12 illustrates an example of a specific operation method of a system providing a voice assistant service according to a first embodiment of the present disclosure. FIG. 13 illustrates an example of a specific operation method of a system providing a voice assistant service according to a second embodiment of the present disclosure. FIG. 14 illustrates an example of a specific operation method of a system providing a voice assistant service according to a third embodiment of the present disclosure. FIG. 15 illustrates an example of a specific operation method of a system providing a voice assistant service according to a fourth embodiment of the present disclosure. FIG. 16 illustrates a block diagram of a voice assistant service providing device according to one embodiment of the present disclosure. FIG. 17 illustrates a specific block diagram of a voice assistant service providing device according to one embodiment of the present disclosure. FIG. 18 illustrates a specific block diagram of a voice assistant service providing device according to one embodiment of the present disclosure. FIG. 19 illustrates a block diagram of a device according to one embodiment of the present disclosure. Specific details for implementing the invention

[0012] Embodiments of the present disclosure are described below in detail with reference to the attached drawings so that those skilled in the art can easily implement them. However, the present disclosure may be embodied in various different forms and is not limited to the embodiments described herein. Furthermore, in order to clearly explain the present disclosure in the drawings, parts unrelated to the explanation have been omitted, and similar parts throughout the specification are denoted by similar reference numerals.

[0013] Some embodiments of the present disclosure may be represented by functional block configurations and various processing steps. Some or all of these functional blocks may be implemented by various numbers of hardware and / or software configurations that execute specific functions. For example, the functional blocks of the present disclosure may be implemented by one or more microprocessors or by circuit configurations for a specific function. Additionally, for example, the functional blocks of the present disclosure may be implemented in various programming or scripting languages. The functional blocks may be implemented as algorithms executed on one or more processors. Furthermore, the present disclosure may employ prior art for electronic configuration, signal processing, and / or data processing, etc.

[0014] Additionally, terms such as "first," "second," etc., may be used to describe various components, but the components should not be limited by these terms. These terms are used for the purpose of distinguishing one component from another.

[0015] Furthermore, the connecting lines or connecting members between the components depicted in the drawings are merely illustrative of functional connections and / or physical or circuit connections. In the actual device, connections between components may be represented by various alternative or added functional connections, physical connections, or circuit connections.

[0016] The present disclosure will be described in detail below with reference to the attached drawings.

[0017] FIG. 1a illustrates a system providing a voice assistant service according to one embodiment of the present disclosure.

[0018] A voice assistant service is a service that provides conversation with a user. A voice assistant service providing device (100) according to one embodiment receives a voice command from a user (10) and, considering the situation of the user (10) and the situation of the voice assistant service providing device (100), can provide a response message to the user as if the user were having a direct conversation with a person, or control another device. In addition, the voice assistant service providing device (100) can appropriately generate and provide information needed by the user, like a personal assistant to the user.

[0019] A voice assistant service providing device (100) according to one embodiment may be a fixed terminal or a mobile terminal implemented as a computer device. The voice assistant service providing device (100) may be, for example, at least one of a smartphone, a mobile phone, a navigation system, a computer, a laptop, a digital broadcasting terminal, a smart speaker, a PDA (Personal Digital Assistants), a PMP (Portable Multimedia Player), and a tablet PC, but is not limited thereto. The voice assistant service providing device (100) may communicate with other devices and / or servers via a network using a wireless or wired communication method.

[0020] As illustrated in FIG. 1, a voice assistant service providing device (100) according to one embodiment may be implemented in the form of a smart speaker that provides a voice assistant service based on a voice command received from a user, but the present disclosure is not limited thereto. A voice assistant service providing device (100) according to one embodiment may be implemented in various forms, such as a device that forms a home network by connecting a plurality of devices, a home edge computer that processes data at the edge of a network, a server that operates a small-scale cloud data center, or a voice assistant server that recognizes a user's voice and controls devices within the home.

[0021] The voice assistant service providing device (100) can interact with multiple devices such as a refrigerator (200-1), a smartphone (200-2), a TV (200-3), a wearable device (200-4), and exercise equipment (200-5). The voice assistant service providing device (100) can provide a response message to a user or control each device by interacting with multiple devices (200-1, 200-2, 200-3, 200-4, 2005-5).

[0022] Conventionally, in order for a user (10) to receive a desired service from a specific device, there was an inconvenience in that the user (10) had to know in advance and speak a command registered in relation to the service of the specific device.

[0023] For example, in order for a user (10) to control the TV (200-3) to play music through the voice assistant service provider (100), the user (10) must utter a voice command such as “TV, play music.” In order to utter such a voice command, the user (10) has the inconvenience of having to know in advance whether the voice assistant service provider (100) is connected to control the TV (200-3), whether the TV (200-3) has a music playback function, and whether playing music through the TV (200-3) is the most suitable among the various devices that have a music playback function.

[0024] In addition, as the number of devices connected to the voice assistant service provider (100) increases, the services that the voice assistant service provider (100) can provide to the user (10) also increase and become more diverse. Therefore, in order for the user (10) to select and receive the desired service, there is a difficulty in having to memorize and speak all of the numerous commands.

[0025] In order to solve these problems, according to various embodiments of the present disclosure, a voice assistant service providing device (100) determines a plurality of devices to which a user's voice command will be transmitted among a plurality of candidate devices connected to the voice assistant service providing device (100), and in response to the voice command, a method is proposed to select at least one service from among the services that can be provided by the plurality of devices and propose it to the user.

[0026] In addition, according to various embodiments of the present disclosure, a method is proposed in which a voice assistant service providing device (100) groups devices according to the capability of the devices in response to a user's voice command, selects an optimal service from among the services that the devices can provide, and proposes it to the user.

[0027] According to various embodiments of the present disclosure, a voice assistant service providing device (100) can activate services that devices can provide based on the context of a user's voice command, group at least some of the devices that suggest the activated services, and recommend and support a service suitable for the user based on the grouped information.

[0028] FIG. 1b illustrates an example of a block diagram of a system providing a voice assistant service according to one embodiment of the present disclosure.

[0029] FIG. 1b illustrates some components of a system including a voice assistant service providing device (100), a first device (200-1), a second device (200-2), and a third device (200-3) according to one embodiment of the present disclosure. FIG. 1b illustrates only some components necessary to explain the respective operations of the voice assistant service providing device (100), the first device (200-1), the second device (200-2), and the third device (200-3). Therefore, the configurations included in the voice assistant service providing device (100), the first device (200-1), the second device (200-2), and the third device (200-3) are not limited to the configurations illustrated in FIG. 1b.

[0030] Referring to FIG. 1b, the voice assistant service providing device (100), the first device (200-1), the second device (200-2), and the third device (200-3) are interconnected via wired or wireless communication and can perform communication.

[0031] In one embodiment, the voice assistant service providing device (100) and a plurality of devices (200-1, 200-2, 200-3) may be directly connected to each other via a communication network, but are not limited thereto. The voice assistant service providing device (100) and the plurality of devices (200-1, 200-2, 200-3) may be connected to a voice assistant server (not shown), and the voice assistant service providing device (100) may be connected to the plurality of devices (200-1, 200-2, 200-3) through the voice assistant server. Additionally, the voice assistant service providing device (100) and the plurality of devices (200-1, 200-2, 200-3) may be connected to an IoT server (not shown). In another embodiment, each of the voice assistant service providing device (100) and the plurality of devices (200-1, 200-2, 200-3) is connected to a voice assistant server through a communication network and can be connected to an IoT server through the voice assistant server.

[0032] For convenience of explanation, the following description uses an example where the operations of the system are performed between a voice assistant service provider (100) and a plurality of devices (200-1, 200-2, 200-3), but the present disclosure is not limited thereto. According to various embodiments of the present disclosure, at least some of the operations of the voice assistant service provider (100) and the plurality of devices (200-1, 200-2, 200-3) described below may be performed on a voice assistant server or an IoT server.

[0033] As illustrated in FIG. 1b, a voice assistant service providing device (100) according to one embodiment may include a voice recognition model (141), a device judgment model (142), a service recommendation model (143), and a response message generation model (145) to provide a voice assistant service to a user.

[0034] The speech recognition model (141) can extract a text interpretation result indicating the user's speech intent through natural language processing of the text converted from the user's voice command.

[0035] The device determination model (142) can determine a plurality of devices to transmit information related to a user's voice command among a plurality of candidate devices connected to the voice assistant service providing device (100). For example, the information related to the user's voice command may include text interpretation results.

[0036] The device determination model (142) can determine the user's intent included in the voice command based on the interpretation result of the text related to the user's voice command. The device determination model (142) can determine the multiple devices among the multiple candidate devices to transmit the voice command based on the relationship between the user's intent and the multiple candidate devices.

[0037] According to one embodiment, a voice assistant service providing device (100) may receive device information from a server including at least one of identification information (e.g., device ID information) for each of a plurality of candidate devices, a device type for each of a plurality of candidate devices, a capability for each of a plurality of candidate devices, location information, and status information. The voice assistant service providing device (100) may use a device determination model (142) to determine a plurality of devices among the plurality of candidate devices to transmit information related to a user's voice command based on the received device information.

[0038] According to another embodiment, a voice assistant service providing device (100) may receive device information directly from a plurality of candidate devices, including at least one of identification information (e.g., device ID information) of each of the plurality of candidate devices, device type of each of the plurality of candidate devices, functional capability of each of the plurality of candidate devices, location information, and status information. The voice assistant service providing device (100) may identify a plurality of candidate devices connected to an access point connected to the voice assistant service providing device (100), and may obtain device information by requesting device information from the identified plurality of candidate devices. The voice assistant service providing device (100) may use a device determination model (142) to determine a plurality of devices among the plurality of candidate devices to transmit information related to a user's voice command based on the obtained device information.

[0039] The voice assistant service providing device (100) can transmit information related to voice commands to a plurality of devices (200-1, 200-2, 200-3) based on a judgment result.

[0040] A plurality of devices (200-1, 200-2, 200-3) that receive information related to a user's voice command may include a voice recognition model (241-1, 241-2, 241-3) for interpreting information related to the user's voice command, and a service provision message generation model (242-1, 242-2, 242-3).

[0041] The speech recognition model (241-1, 241-2, 241-3) can extract the user's speech intent by performing natural language understanding processing on information related to the user's voice command.

[0042] Each of the plurality of devices (200-1, 200-2, 200-3) can determine whether it can provide a service in response to information related to a received voice command. If a device determines that it can provide a service in response to information related to a voice command, the service provision message generation model (242-1, 242-2, 242-3) can generate a service provision message to be transmitted to a voice assistant service provision device (100). The service provision message generation model (242-1, 242-2, 242-3) can determine a service suitable for the user's voice command and generate and transmit a service provision message of a predetermined format based on the determined service.

[0043] The service recommendation model (143) of the voice assistant service providing device (100) can select at least one service providing message most suitable for the user from among a plurality of service providing messages received from a plurality of devices (200-1, 200-2, 200-3).

[0044] The response message generation model (145) can convert at least one selected service provision message into human-understandable natural language and convert the text-based natural language into a speech signal and output it.

[0045] Hereinafter, with reference to FIGS. 2 and FIGS. 3, an example of the overall operation method of a system providing a voice assistant service according to an embodiment of the present disclosure will be described.

[0046] FIG. 2 illustrates a signal flow diagram of an operation method of a system providing a voice assistant service according to one embodiment of the present disclosure. FIG. 2 illustrates, as an example, a case in which a voice assistant service providing device (100) according to one embodiment interacts with a plurality of devices (200-1, 200-2, 200-3).

[0047] A voice assistant service providing device (100) according to one embodiment receives a first voice command from a user (S210). The voice assistant service providing device (100) can receive the user's first voice command through a microphone. The voice assistant service providing device (100) can determine a plurality of devices (200-1, 200-2, 200-3) to transmit information related to the user's first voice command among a plurality of candidate devices connected to the voice assistant service providing device (100).

[0048] The voice assistant service providing device (100) can determine the user's intent through natural language understanding processing for the first voice command. The voice assistant service providing device (100) can determine multiple devices (200-1, 200-2, 200-3) among multiple candidate devices based on the relationship between the user's intent and multiple candidate devices.

[0049] For example, a voice assistant service providing device (100) can obtain information about multiple candidate devices from an external server, internal memory, or multiple candidate devices, and determine the relationship between the user's intention and the multiple candidate devices based on the obtained device information.

[0050] A voice assistant service providing device (100) according to one embodiment transmits information related to a first voice command to a plurality of devices (200-1, 200-2, 200-3) based on a judgment result (S220).

[0051] Information related to a voice command transmitted to multiple devices (200-1, 200-2, 200-3) may include an audio signal of the user's voice command itself or a result of processing the voice command. A voice assistant service providing device (100) may convert the first voice command into a first text by performing voice recognition on the first voice command, extract a speech intent by interpreting the first text through natural language understanding processing, and transmit the result to multiple devices (200-1, 200-2, 200-3).

[0052] Each of the plurality of devices (200-1, 200-2, 200-3) can determine whether to provide a service in response to information related to the received first voice command and learn the result of the determination. For example, each device can determine whether to provide a service in response to information related to the received first voice command based on at least one of information about services that the device can provide, information about devices constituting the device, the type of service requested by the first voice command, current time information, and location information of the device.

[0053] Each of the plurality of devices (200-1, 200-2, 200-3) can learn information related to a first voice command received from a voice assistant service provider (100), information determining whether the device can provide a service in response to the first voice command, and situational information (e.g., user location information, location information of the voice assistant service provider (100), location information of the device, current time information, etc.). If the device can learn on its own, it can learn information related to the voice command on its own. If it cannot learn on its own, the device can learn information related to the voice command by connecting with a server (or cloud).

[0054] For example, as illustrated in FIG. 2, in step S231, the first device (200-1) may determine that it can provide a service in response to the first voice command. In step S232, the second device (200-2) may determine that it cannot provide a service in response to the first voice command. In step S233, the third device (200-3) may determine that it can provide a service in response to the first voice command.

[0055] If the device determines that it can provide a service in response to information related to the first voice command, the device may generate a service provision message to be transmitted to the voice assistant service providing device (100). The service provision message refers to a message output by the device in response to information related to the user's voice command received from the voice assistant service providing device (100).

[0056] For example, a service provision message may include at least one of whether the device can provide a service that responds to a voice command, whether the device is included in a predetermined group related to the voice command, identification information of the device, identification information of a predetermined service provided by the device in response to the voice command, the type of the predetermined service, identification information of an application used to provide the predetermined service, and result text related to the service proposed by the device.

[0057] In step S241 of FIG. 2, the first device (200-1) can generate a service provision message in response to information related to the first voice command and transmit it to the voice assistant service provision device (100). In step S243 of FIG. 2, the third device (200-3) can generate a service provision message in response to information related to the first voice command and transmit it to the voice assistant service provision device (100).

[0058] According to one embodiment of the present disclosure, a device connected to a voice assistant service providing device (100) can determine a service suitable for a user's voice command and, based on the determined service, generate and transmit a service providing message of a predetermined format. By transmitting the service providing message, the device can propose to the voice assistant service providing device (100) a service determined to be suitable for the user's voice command.

[0059] In addition, the first device (200-1) and the third device (200-3) may further transmit at least one of the service provision history information of the device and the device information of the device, in addition to the service provision message responding to the first voice command.

[0060] Service provision history information refers to information regarding past history related to services provided or proposed by the device in response to information related to a user's specific voice command received from the voice assistant service providing device (100). For example, the service provision history information may include at least one of whether the device can provide a service that responds to a voice command, the number of times a specific service proposed by the device in response to a voice command was selected by the user, and situational information (e.g., time information, location information, etc.) when the specific service was selected in response to a voice command.

[0061] Device information refers to information about the devices constituting the device. Specifically, device information may include information regarding whether the device includes an input / output interface for receiving user input from a user or providing information to a user, and information related to the performance of each input / output interface. For example, device information may include information regarding whether the device includes a display, the resolution supported by the display, whether the device includes a microphone, the sound quality supported by the microphone, whether the device includes a speaker, the sound quality supported by the speaker, and at least one of the fields of services that the device can provide.

[0062] In step S250, the voice assistant service providing device (100) can group the devices based on service providing messages received from the devices. The voice assistant service providing device (100) can determine the devices capable of providing a service that responds to the first voice command as a first group related to the first voice command. Referring to FIG. 2, the voice assistant service providing device (100) determines the first device (200-1) and the third device (200-3) that transmitted the service providing message as the first group, and can transmit a message to the first device (200-1) and the third device (200-3) designating the devices as the first group (S260).

[0063] In step S270, the voice assistant service providing device (100) can select at least one service providing message and output a response message based on the result that the first device (200-1) and the third device (200-3) are designated as one group.

[0064] The voice assistant service providing device (100) can select at least one service providing message from among the service providing messages of the first device (200-1) and the third device (200-3) determined as the first group. The voice assistant service providing device (100) can generate and output a response message based on the selected at least one service providing message.

[0065] A voice assistant service providing device (100) can select at least one of a plurality of service providing messages by using a service recommendation model, which is an artificial intelligence model. The service recommendation model may be an artificial intelligence algorithm learned by a first voice command, a service providing history of a plurality of devices for the first voice command, and device information of a plurality of devices.

[0066] As an example, a voice assistant service providing device (100) can select one service providing message most suitable for the user from among service providing messages of a first device (200-1) and a third device (200-3) determined as a first group, and output a response message based on the selected service providing message.

[0067] For example, a voice assistant service provider (100) may receive a voice command saying “play music” and transmit information related to the voice command to connected devices. In response to the voice command, the voice assistant service provider (100) may receive service provision messages including information that music can be played from a TV and a smartphone, and information regarding speaker performance for music playback. Based on the device information of the TV and the smartphone, the voice assistant service provider (100) may select a service provision message received from a TV with better speaker performance. Based on the selected service provision message, the voice assistant service provider (100) may output a response message saying “I will play music through the TV” and control the TV to play music.

[0068] As another example, the voice assistant service providing device (100) can combine service providing messages from the first device (200-1) and the third device (200-3) determined as the first group to generate and output a single response message. For example, the voice assistant service providing device (100) can combine service providing messages received from a TV and a smartphone to output a response message saying, “You can play music through the TV or play music through the smartphone.”

[0069] The user can listen to the response message and decide whether to accept the service proposed by the voice assistant service provider (100). In step S280, the voice assistant service provider (100) can receive a second voice command from the user. Based on the second voice command, the voice assistant service provider (100) can determine whether the user has decided to accept the service proposed by the voice assistant service provider (100).

[0070] In step S290, the voice assistant service providing device (100) can determine the service selected by the user.

[0071] The voice assistant service providing device (100) can determine whether a service recommended by the voice assistant service providing device (100) to the user has been selected, or whether a service other than the service recommended by the voice assistant service providing device (100) has been selected.

[0072] The voice assistant service providing device (100) can transmit information related to a selected service to a first device (200-1) and a third device (200-3) determined as a first group (S291). The first device (200-1) and the third device (200-3) can learn information related to the selected service (S293, S295). Based on information related to the service selected by the user, the first device (200-1) and the third device (200-3) can update the 'number of times the service was selected by the user' or the 'number of times the service was not selected by the user' included in the service provision history information. By learning the service selected by the user, the devices can suggest the optimal service preferred by the user.

[0073] As illustrated in FIG. 2, a voice assistant service providing device (100) according to one embodiment can group devices related to a first voice command into a first group based on service providing messages received from devices. The first device (200-1) and the third device (200-3) can learn that the first device (200-1) and the third device (200-3) are included in the first group related to the first voice command based on the message received in step S260. A voice assistant service providing system that has learned grouping information related to the first voice command can operate as illustrated in FIG. 3.

[0074] FIG. 3 illustrates a signal flow diagram of an operation method of a system providing a voice assistant service according to an embodiment of the present disclosure. FIG. 3 illustrates, as an example, the case in which a fourth device (200-4) is added to the voice assistant service providing system shown in FIG. 2. The voice assistant service providing device (100) shown in FIG. 3 interacts with a plurality of devices (200-1, 200-2, 200-3, 200-4).

[0075] In step S310 of FIG. 3, the voice assistant service providing device (100) according to one embodiment may receive a voice command identical to the first voice command received in step S210 of FIG. 2. The voice assistant service providing device (100) may receive the user's first voice command through a microphone. The voice assistant service providing device (100) may determine a plurality of devices (200-1, 200-2, 200-3, 200-4) to transmit information related to the user's first voice command among a plurality of candidate devices connected to the voice assistant service providing device (100).

[0076] A voice assistant service providing device (100) according to one embodiment transmits information related to a first voice command to a plurality of devices (200-1, 200-2, 200-3, 200-4) based on a judgment result (S320).

[0077] Information related to a voice command transmitted to multiple devices (200-1, 200-2, 200-3, 200-4) may include an audio signal of the user's voice command itself or a result of processing the voice command. A voice assistant service providing device (100) may convert a first voice command into a first text by performing voice recognition on a first voice command, extract a speech intent by interpreting the first text through natural language understanding processing, and transmit this to multiple devices (200-1, 200-2, 200-3, 200-4).

[0078] Each of the plurality of devices (200-1, 200-2, 200-3, 200-4) can determine whether it can provide a service in response to information related to the received first voice command. For example, as illustrated in FIG. 3, at step S331, the first device (200-1) can determine that it can provide a service in response to the first voice command. At step S332, the second device (200-2) can determine that it cannot provide a service in response to the first voice command. At step S333, the third device (200-3) can determine that it can provide a service in response to the first voice command. At step S334, the fourth device (200-4) can determine that it can provide a service in response to the first voice command.

[0079] If a device determines that it can provide a service in response to information related to a first voice command, the device may generate a service provision message to be transmitted to a voice assistant service providing device (100). The first device (200-1), the third device (200-3), and the fourth device (200-4) may generate a service provision message in response to information related to a first voice command and transmit it to the voice assistant service providing device (100) (S341, S343, S344).

[0080] The first device (200-1) has learned that it is included in the first group associated with the first voice command based on the message received in step S260 of FIG. 2. Accordingly, the service provision message transmitted by the first device (200-1) in step S341 of FIG. 3 may include information that the first device (200-1) is included in the first group associated with the first voice command. The third device (200-3) has also learned that it is included in the first group associated with the first voice command based on the message received in step S260. Accordingly, the service provision message transmitted by the third device (200-3) in step S343 may include information that the third device (200-3) is included in the first group associated with the first voice command. However, since the fourth device (200-4) has been newly added, it has not learned group information related to the first voice command. Therefore, the service provision message transmitted by the fourth device (200-4) in step S344 may include information that the fourth device (200-4) is not included in the first group related to the first voice command.

[0081] In step S350, the voice assistant service providing device (100) can select at least one service providing message based on group information and output a response message.

[0082] The voice assistant service providing device (100) can determine that the first device (200-1) and the third device (200-3) are included in a first group based on service providing messages transmitted by the first device (200-1), the third device (200-3), and the fourth device (200-4). The voice assistant service providing device (100) can select at least one service providing message from among the service providing messages of the first device (200-1) and the third device (200-3) determined to be in the first group. The voice assistant service providing device (100) can generate and output a response message based on the selected at least one service providing message.

[0083] In FIG. 3, the voice assistant service providing device (100) can obtain group information based on service providing messages received from devices without performing an operation to group devices by analyzing all data received from devices. Accordingly, according to the operation method illustrated in FIG. 3 compared to FIG. 2, the voice assistant service providing device (100) can select at least one service providing message and output a response message more quickly.

[0084] Although FIG. 3 illustrates an example in which a voice assistant service provider (100) operates in connection with four devices, in reality, it can operate in connection with a much larger number of devices. As the voice assistant service provider (100) operates in connection with a larger number of devices, the amount of data received from the devices (e.g., service provision messages, service provision history, and device information, etc.) increases. Consequently, it takes a long time for the voice assistant service provider (100) to analyze all of the large amount of data and group at least some of the devices into a group related to voice commands. However, as illustrated in FIG. 3, the voice assistant service provider (100) according to one embodiment can group devices more quickly by obtaining group information that the devices have learned in advance through a service provision message, without analyzing all the data received from the devices.

[0085] After outputting a response message in step S350, in step S360, the voice assistant service providing device (100) can regroup the devices by analyzing the service providing messages, service providing history, and device information received from the devices.

[0086] A voice assistant service providing device (100) may determine devices capable of providing a service that responds to a first voice command as a first group related to the first voice command. Referring to FIG. 3, the voice assistant service providing device (100) may update group information to include a fourth device (200-4) in the first group that included a first device (200-1) and a third device (200-3). The voice assistant service providing device (100) may transmit the updated group information to the first device (200-1), the third device (200-3), and the fourth device (200-4) (S370).

[0087] The user can listen to the response message and decide whether to accept the service proposed by the voice assistant service provider (100). In step S380, the voice assistant service provider (100) may receive a second voice command from the user. Based on the second voice command, the voice assistant service provider (100) may determine whether the user has decided to accept the service proposed by the voice assistant service provider (100).

[0088] In step S390, the voice assistant service providing device (100) can determine the service selected by the user.

[0089] The voice assistant service providing device (100) can determine whether a service recommended by the voice assistant service providing device (100) to the user has been selected, or whether a service other than the service recommended by the voice assistant service providing device (100) has been selected.

[0090] The voice assistant service providing device (100) can transmit information related to a selected service to a first device (200-1), a third device (200-3), and a fourth device (200-4) determined as a first group (S391). The first device (200-1), the third device (200-3), and the fourth device (200-4) can learn information related to the selected service (S393, S394, S395). Based on information related to the service selected by the user, the first device (200-1), the third device (200-3), and the fourth device (200-4) can update the 'number of times the service was selected by the user' or the 'number of times the service was not selected by the user' included in the service provision history information. By learning the service selected by the user, the devices can suggest the optimal service preferred by the user.

[0091] As described above, a voice assistant service providing device (100) according to one embodiment of the present disclosure can provide fast and accurate conversational services by grouping devices according to the capabilities of the devices, selecting an optimal service from among the services that the devices included in the group can provide, and suggesting it to the user.

[0092] Below, we will examine in detail how the voice assistant service providing device (100) operates to provide the service.

[0093] FIG. 4 illustrates a flowchart of a method for providing a voice assistant service according to one embodiment of the present disclosure.

[0094] In step S410, a voice assistant service providing device (100) according to one embodiment of the present disclosure can receive a first voice command from a user.

[0095] The voice assistant service providing device (100) can determine which of the multiple candidate devices connected to the voice assistant service providing device (100) will transmit information related to the user's first voice command.

[0096] The voice assistant service providing device (100) can convert a first voice command into a first text and interpret the converted first text. The voice assistant service providing device (100) can determine a plurality of devices related to the interpreted text among a plurality of candidate devices using a device determination model.

[0097] According to one embodiment, a voice assistant service providing device (100) can analyze text using a natural language understanding model, determine a user's intention based on the analysis result of the text, and determine a plurality of devices among a plurality of candidate devices that can provide a service according to the user's intention. The voice assistant service providing device (100) can determine a plurality of devices among a plurality of candidate devices to transmit a first voice command based on the relationship between the user's intention and the plurality of candidate devices.

[0098] For example, a voice assistant service providing device (100) may acquire device information regarding a plurality of candidate devices and, based on the device information, acquire a plurality of probability values ​​indicating the degree of correlation between the user's intention and the plurality of candidate devices. The voice assistant service providing device (100) may determine among the plurality of candidate devices that have probability values ​​greater than or equal to a threshold value.

[0099] A voice assistant service providing device (100) can determine multiple devices related to a user's intention recognized from text based on a matching model that determines the relationship between the user's intention and multiple candidate devices. The voice assistant service providing device (100) can determine multiple devices among multiple candidate devices that have a high probability of providing a service according to the user's intention through a device determination model.

[0100] According to one embodiment, a voice assistant service providing device (100) can obtain multiple probability values ​​indicating the degree of correlation between a user's intention and multiple candidate devices by applying a matching model to a user's intention. The voice assistant service providing device (100) can decide to transmit information related to a voice command to multiple devices having a high value among the obtained multiple probability values.

[0101] As an example, the voice assistant service providing device (100) can determine, through a device determination model, a predetermined number of multiple devices among multiple candidate devices that have a high probability of providing a service according to the user's intention. As another example, the voice assistant service providing device (100) can determine, through a device determination model, all devices among multiple candidate devices that have a probability of providing a service according to the user's intention that is greater than or equal to a threshold value as multiple devices to transmit information related to voice commands.

[0102] The voice assistant service providing device (100) may, for example, learn a matching model between the user's intent and candidate devices using a rule-based system, but is not limited thereto. The artificial intelligence model used by the voice assistant service providing device (100) may, for example, be a neural network-based system (e.g., convolutional neural network (CNN), recurrent neural network (RNN), Support Vector Machine (SVM), linear regression, logistic regression, Naive Bayes classification (Na It may be a random forest, a decision tree, or a k-nearest neighbor algorithm. Or, it may be a combination of the above or other artificial intelligence models.

[0103] Meanwhile, a voice assistant service providing device (100) according to one embodiment may decide to transmit the first voice command to a predetermined device among a plurality of candidate devices when a predetermined device is specified in the user's first voice command, a request for a service of the predetermined device is specified, or information related to the predetermined device is included. Alternatively, the voice assistant service providing device (100) may decide to transmit the first voice command to a plurality of devices associated with the predetermined device among a plurality of candidate devices. In such cases, when the target device requesting the service is clearly specified in the voice command, the voice assistant service providing device (100) can simply determine the device to which the voice command will be transmitted without using a device determination model.

[0104] In step S420, a voice assistant service providing device (100) according to one embodiment of the present disclosure may transmit information related to a first voice command to a plurality of devices. The voice assistant service providing device (100) may broadcast information related to a first voice command to a plurality of devices.

[0105] The "information related to the first voice command" transmitted to multiple devices may include the audio signal of the user's voice command itself or the result of processing the voice command. For example, the result of processing the voice command may include at least one of a feature vector extracted from the user's voice command, text converted from the user's voice command, a result output after natural language understanding (NLU) processing of the text, a type of service requested by the voice command determined based on the result of interpreting the text, the volume of the user's voice command, and the SNR of the user's voice command.

[0106] The voice assistant service providing device (100) can convert a first voice command into a first text and interpret the converted first text. The voice assistant service providing device (100) can transmit information regarding the result of interpreting the first text and the type of service requested by the first voice command.

[0107] The voice assistant service providing device (100) can determine whether the type of service requested by the first voice command is a notification or an action based on the result of interpreting the first text.

[0108] The voice assistant service providing device (100) may determine that the type of service requested by the first voice command is "performance of operation" when it is determined that the first voice command requests the device to perform a predetermined operation (e.g., playing media, recording, turning the power on or off). For example, when a user's voice command "play music" is received, the voice assistant service providing device (100) may determine that the type of service requested by the voice command is "performance of operation".

[0109] The voice assistant service providing device (100) may determine that the type of service requested by the first voice command is "notification" when it is determined that the first voice command does not request the device to perform a predetermined operation. For example, when a user's voice command "I'm hungry, what should I eat?" is received, the voice assistant service providing device (100) may determine that the type of service requested by the voice command is "notification".

[0110] In step S430, a voice assistant service providing device (100) according to one embodiment of the present disclosure can receive a plurality of service providing messages responding to a first voice command from each of a plurality of devices and service providing history information of the plurality of devices.

[0111] A "service provision message" received from each device may include at least one of whether the device provides a service in response to a first voice command, whether the device is included in a first group, identification information of the device, identification information of a first service provided by the first device in response to a first voice command, the type of the first service, and identification information of an application used to provide the first service.

[0112] The "service provision history information" received from each device may include at least one of whether the device provides a service that responds to the first voice command, the number of times the first service proposed by the first device for the first voice command was selected by the user, the number of times the first service proposed by the first device for the first voice command was not selected by the user, and situation information when the first service was selected for the first voice command.

[0113] The voice assistant service providing device (100) can further receive device information constituting each device, along with service providing messages and service providing history information from a plurality of devices.

[0114] “Device information” means information about the devices constituting the device. Specifically, the device information may include information regarding whether the device includes an input / output interface for receiving user input or providing information to the user, and information related to the performance of each input / output interface. For example, the device information may include information regarding whether the device includes a display, the resolution supported by the display, whether the device includes a microphone, the sound quality supported by the microphone, whether the device includes a speaker, the sound quality supported by the speaker, and at least one of the fields of services that the device can provide.

[0115] In step S440, a voice assistant service providing device (100) according to one embodiment of the present disclosure may select at least one of a plurality of service providing messages based on service providing history information.

[0116] A voice assistant service providing device (100) can identify devices included in a first group among a plurality of devices based on service providing messages and / or service providing history.

[0117] As an example, a voice assistant service providing device (100) may analyze service provision history information received from multiple devices when multiple devices have not learned group information related to a first voice command, and determine devices included in a first group corresponding to a user's first voice command based on the analysis result. The voice assistant service providing device (100) may obtain information (isSupported data) regarding whether each device can provide a service that responds to the first voice command from the service provision history information.

[0118] As another example, the voice assistant service providing device (100) can identify devices included in the first group based on the group information included in the service providing messages received from the plurality of devices when the plurality of devices have learned group information related to the first voice command.

[0119] A voice assistant service providing device (100) can select at least one of the service providing messages received from the identified devices based on service providing history information of the devices identified as being included in the first group.

[0120] The voice assistant service providing device (100) can select at least one of a plurality of service providing messages by further considering device information of a plurality of devices in addition to service providing history information.

[0121] A voice assistant service providing device (100) can select at least one of a plurality of service providing messages using a service recommendation model. The service recommendation model may be an artificial intelligence algorithm learned by a predetermined voice command, the service provision history of a plurality of devices regarding the predetermined voice command, and device information of each of the plurality of devices. The voice assistant service providing device (100) is trained to select and recommend the service most preferred by the user among the services proposed by the devices.

[0122] As an example, a voice assistant service providing device (100) may assign priority or weight to a plurality of service providing messages based on service providing history information and / or device information in order to select at least one service providing message. The voice assistant service providing device (100) may assign priority or weight to a plurality of service providing messages based on at least one of whether each device provides a service in response to a first voice command, the type of service provided by each device in response to the first voice command, the number of times the service provided by each device was selected for the first voice command, situation information when the service provided by each device was selected for the first voice command, and device information constituting each device. The voice assistant service providing device (100) may select one of the plurality of service providing messages based on the assigned priority or weight.

[0123] As another example, a voice assistant service providing device (100) may select two or more service providing messages among a plurality of service providing messages based on priority or weight assigned according to service providing history information and / or device information.

[0124] In step S450, the voice assistant service providing device (100) according to one embodiment can output a response message that responds to the first voice command based on the selected result.

[0125] Multiple service provision messages received from multiple devices may include information related to a service provided by each of the multiple devices in response to a first voice command. The voice assistant service providing device (100) may generate a response message to include information related to at least one service within at least one selected service providing message. The voice assistant service providing device (100) may output a response message.

[0126] Meanwhile, the voice assistant service providing device (100) according to one embodiment may omit the execution of steps S410 to S440 if a specific device is specified within the user's first voice command, a request for a service of the specific device is specified, or information related to the specific device is included. The voice assistant service providing device (100) may transmit information related to the first voice command to the specific device, receive a service providing message from the specific device, and generate and output a response message that includes information related to the service within the received service providing message. In this way, when the target device requesting the service is clearly specified in the voice command, the voice assistant service providing device (100) may simply generate a response message without using a device judgment model and / or a service recommendation model.

[0127] FIG. 5 illustrates a flowchart that specifies step S440 of FIG. 4 in a specific method in which a voice assistant service providing device (100) provides a voice assistant service according to one embodiment. Accordingly, descriptions that overlap with FIG. 4 regarding steps S410 to S430 are omitted.

[0128] In step S541, a voice assistant service providing device (100) according to one embodiment can identify devices included in a first group among a plurality of devices. The voice assistant service providing device (100) can identify devices included in the first group based on service providing messages or service providing history information received from a plurality of devices.

[0129] For example, a voice assistant service providing device (100) may obtain information from a service providing message regarding whether a device is included in a first group associated with a first voice command, and determine that the device is included in the first group based on the obtained information.

[0130] Alternatively, for example, the voice assistant service providing device (100) may obtain information from service provision history information regarding whether the device can provide a service that responds to the first voice command, and determine that the device is included in the first group based on the obtained information. The voice assistant service providing device (100) may obtain information (e.g., isSupported flag) regarding whether the device can provide a service that responds to the first voice command from the service provision history information.

[0131] In step S543, a voice assistant service providing device (100) according to one embodiment can select at least one of the service providing messages received from the identified devices based on the service providing history information of the devices identified as being included in the first group.

[0132] The voice assistant service providing device (100) may further consider device information of multiple devices when selecting at least one service providing message.

[0133] According to one embodiment, the voice assistant service providing device (100) can select all service providing messages received from the first group of devices.

[0134] According to another embodiment, the voice assistant service providing device (100) may assign priority or weight to a first group of service providing messages based on service providing history information and / or device information, and select at least one service providing message according to the assigned priority or weight. As an example, the voice assistant service providing device (100) may select one service providing message assigned the highest priority or select multiple service providing messages assigned relatively high priority. As another example, the voice assistant service providing device (100) may select at least one service providing message in which the assigned weight is greater than or equal to a threshold value.

[0135] In step S450, the voice assistant service providing device (100) according to one embodiment can output a response message that responds to the first voice command based on the result selected in step S543.

[0136] A voice assistant service providing device (100) according to one embodiment may generate a response message that includes information related to at least one service within at least one selected service providing message. The voice assistant service providing device (100) may generate a response message through a Natural Language Generation process that converts at least one selected service providing message into a human-understandable natural language and a Text-to-Speech (TTS) process that converts the natural language in text form into a voice signal.

[0137] A voice assistant service providing device (100) according to one embodiment can generate and output one response message by listing the selected service providing messages when two or more service providing messages are selected in step S543.

[0138] A voice assistant service providing device (100) according to another embodiment can combine the selected service providing messages to generate and output one response message when two or more service providing messages are selected in step S543.

[0139] The user can listen to a response message responding to a first voice command and decide whether to accept a service recommended by the voice assistant service provider (100). Based on the decision, the user can utter a second voice command.

[0140] In step S560, a voice assistant service providing device (100) according to one embodiment of the present disclosure can receive a second voice command from a user.

[0141] In step S570, a voice assistant service providing device (100) according to one embodiment can determine whether the user has decided to accept a service recommended by the voice assistant service providing device (100) based on a second voice command received from the user. Based on the second voice command, the voice assistant service providing device (100) can determine a service selected by the user among services provided by a plurality of devices.

[0142] In step S580, the voice assistant service providing device (100) can transmit information related to a service selected by the user to a plurality of devices. The voice assistant service providing device (100) can transmit information related to a service selected by the user to devices identified as being included in the first group in step S541.

[0143] In step S590, the voice assistant service providing device (100) according to one embodiment can learn a service recommendation model based on information related to a service selected by the user. The voice assistant service providing device (100) can continuously learn a service recommendation model by the user's voice command in order to suggest an optimal service preferred by the user among various services provided by multiple devices.

[0144] Referring to FIG. 5, a voice assistant service providing device (100) according to one embodiment determines a service selected by a user based on a second voice command of the user. However, various embodiments of the present disclosure are not limited to the example shown in FIG. 5.

[0145] A voice assistant service providing device (100) according to another embodiment can determine a service selected by a user by detecting a user's action or receiving user input. For example, a device capable of receiving touch input from a user, such as a smartphone or a wearable device, can determine whether the user has selected a proposed service based on the user's touch input.

[0146] For example, in response to a first voice command, “What should I eat?”, the voice assistant service provider (100) may output a response message, “There is pork belly in the refrigerator.” If the user decides to accept the service suggested by the voice assistant service provider (100), the user may take the pork belly out of the refrigerator. Based on the user’s action of taking the pork belly out of the refrigerator, the voice assistant service provider (100) may determine that the user has decided to accept the service suggested by the voice assistant service provider (100).

[0147] Meanwhile, a voice assistant service providing device (100) according to one embodiment may obtain device information from a plurality of devices connected to the voice assistant service providing device (100). For example, the device information may include resource information and / or capability information. The resource information of the device may include information about components required for the operation of the device, such as a memory device, an input / output device, or a central processing unit. The capability information of the device may include information about functions that the device can perform, version information of software running on the device, or information related to the performance of the device's components. For example, when a new device is additionally connected, the voice assistant service providing device (100) may obtain device information about the device from the newly connected device.

[0148] A voice assistant service providing device (100) according to one embodiment can perform more operations than those described above with reference to FIG. 4 based on device information of a plurality of devices, and can play a more leading role in providing voice assistant services. That is, at least some of the operations and / or functions of the device described in the present disclosure can be performed by the voice assistant service providing device (100).

[0149] For example, if the resources of a specific device (e.g., capacity of a central processing unit or memory storage space, etc.) are insufficient, the specific device may not be able to manage service provision history information. The voice assistant service providing device (100) may determine, based on device information received from the specific device, that it is difficult to receive service provision history information from the specific device and that only service provision messages can be received. Alternatively, the voice assistant service providing device (100) may determine, based on device information received from the specific device, that only a portion of the service provision history information can be received from the specific device.

[0150] In such cases, since it is not possible to determine whether a specific device is included in a group related to voice commands based on service provision history information, as an example, the voice assistant service providing device (100) may determine that the specific device is included in a group related to voice commands by default. Additionally, since the voice assistant service providing device (100) cannot consider the service provision history information of the specific device when selecting a service provision message, the service provision message of the specific device among a plurality of service provision messages may be set as a default value that is always selected. As another example, the voice assistant service providing device (100) may directly manage the service provision history information of a specific candidate device.

[0151] Below, we will specifically examine how a device interacting with a user through a voice assistant service providing device (100) provides information related to the service to the voice assistant service providing device (100).

[0152] FIG. 6 illustrates a flowchart of a method of operation of a device interacting with a user through a voice assistant service providing device according to one embodiment of the present disclosure.

[0153] In step S610, the device (200) according to one embodiment can receive information related to the first voice command from the voice assistant service providing device (100).

[0154] Information related to a first voice command received by the device (200) from the voice assistant service provider (100) may include an audio signal of the user's voice command itself or a result of processing the voice command. For example, the result of processing the voice command may include at least one of a feature vector extracted from the user's voice command, text converted from the user's voice command, a result of the text being output after undergoing natural language understanding (NLU) processing, a type of service requested by the voice command determined based on the result of interpreting the text, the volume of the user's voice command, and the SNR of the user's voice command.

[0155] For example, the device (200) may receive information regarding the result of interpreting the first text converted from the first voice command and the type of service requested by the first voice command as information related to the first voice command.

[0156] In step S620, the device (200) according to one embodiment can determine whether the device (200) can provide a service in response to information related to the first voice command.

[0157] A device (200) according to one embodiment can identify services that the device (200) can provide based on information about devices constituting the device (200) (e.g., screen, input interface, etc.), software information such as an application running on the device (200), and information stored in advance in the device (200) (e.g., current time information and location information of the device, etc.).

[0158] The information stored in advance in the device (200) may include information necessary for learning to determine the services that the device (200) can provide. For example, the information stored in advance in the device (200) may include service provision history information of the device (200).

[0159] If the device (200) includes a service corresponding to the type of service requested by the first voice command among the services identified as being available to the device (200), the device (200) may select one service corresponding to the type of service requested by the first voice command among the services identified as being available to the device (200) and generate a service provision message.

[0160] A device (200) according to one embodiment performs machine learning to determine a supported service and can increase the accuracy of the service proposed to the user. Below, we will specifically examine a method for the device (200) to learn the capability of the device (200) related to the supported service.

[0161] Initially, each device (200) linked with the voice assistant service provider (100) can learn information about the services that the device can support using incremental learning. For the initial learning, general information such as the device's location information and time information, and the device's device information may be used.

[0162] After the initial training, transfer learning based on the trained model can be utilized. Each device can improve the accuracy of understanding the context of voice commands and the accuracy of services proposed in response to voice commands by training only the final linear classifier layer of the trained model. For example, for transfer learning, only information related to the service selected by the user regarding the user's voice commands and proposed services may be used.

[0163] In addition, each group generated by grouping devices can be classified by a classifier using the Mixture of Experts method of ensemble learning, thereby increasing the accuracy of the proposed service. Each device can also increase the accuracy of the proposed service by learning a gating function based on the device's capabilities in relation to the region of each classifier. Furthermore, each device can increase the accuracy of the proposed service by learning information that there is no service available in response to a specific voice command, or information that the proposed service was not selected by the user in response to a specific voice command.

[0164] If the device (200) determines that it can provide a service in response to information related to a first voice command, the device (200) may generate a service provision message to be transmitted to a voice assistant service providing device (100). The service provision message may include information related to a service proposed by the device (200) in response to information related to a first voice command received from the voice assistant service providing device (100).

[0165] For example, a service provision message may include at least one of whether the device can provide a service that responds to a voice command, whether the device is included in a predetermined group associated with the voice command, identification information of the device, identification information of a predetermined service provided by the device in response to a voice command, the type of the predetermined service, and identification information of an application used to provide the predetermined service.

[0166] In step S631, the device (200) according to one embodiment can transmit a service provision message and service provision history information.

[0167] Service provision history information refers to information regarding past history related to services provided or proposed by the device (200) in response to information related to the first voice command. For example, the service provision history information may include at least one of whether the device can provide a service in response to the voice command, the number of times a predetermined service proposed by the device in response to the voice command was selected by the user, and situational information (e.g., time information, location information, etc.) when the predetermined service was selected in response to the voice command.

[0168] In one embodiment, when the device (200) determines that it can provide a service in response to information related to the first voice command, it may record and transmit a value corresponding to "Yes" in the isSupported flag included in the service provision history information.

[0169] Meanwhile, if the device (200) determines that it cannot provide a service in response to information related to the first voice command, the device (200) may transmit a message indicating that service provision is impossible in step S633.

[0170] For example, the device (200) may record a value corresponding to "No" in the isSupported flag and transmit it to the voice assistant service provider device (100). However, the embodiment is not limited to the flowchart shown in FIG. 6, and the device (200) may not transmit a message indicating that service provision is impossible when it is impossible to provide a service in response to the first voice command, and may not perform any action.

[0171] FIG. 7 illustrates a flowchart of a method in which a device (200) learns user feedback regarding a proposed service in addition to the operation method shown in FIG. 6.

[0172] FIG. 7 illustrates a specific flowchart of a method of operation of a device interacting with a user through a voice assistant service providing device according to an embodiment of the present disclosure. Descriptions that overlap with FIG. 6 regarding steps S610 and S620 are omitted.

[0173] When the device (200) transmits a service provision message and service provision history information in step S631, the voice assistant service providing device (100) according to one embodiment can select at least one of a plurality of service provision messages based on the service provision history information.

[0174] The voice assistant service providing device (100) can identify devices included in a first group among a plurality of devices. The voice assistant service providing device (100) can select at least one of the service providing messages received from the identified devices based on the service providing history information of the devices identified as being included in the first group.

[0175] A voice assistant service providing device (100) can select at least one of a plurality of service providing messages based on at least one of device information and service providing history information of a plurality of devices. The voice assistant service providing device (100) can output a response message that responds to a first voice command based on the selected service providing message.

[0176] The user can listen to a response message responding to a first voice command and decide whether to accept a service proposed by the voice assistant service provider (100). Based on the decision, the user can utter a second voice command.

[0177] A voice assistant service providing device (100) according to one embodiment may receive a second voice command from a user. Based on the second voice command received from the user, the voice assistant service providing device (100) may determine whether the user has decided to accept a service proposed by the voice assistant service providing device (100). Based on the second voice command, the voice assistant service providing device (100) may determine a service selected by the user among services provided by a plurality of devices.

[0178] In step S740, the device (200) can receive information related to a service selected by the user from the voice assistant service provider device (100).

[0179] In step S750, the device (200) can update the service provision history information based on the received information. For example, if the service proposed through the service provision message transmitted in step S631 is selected by the user, the device (200) can increase the Pick count of the service provision history information by 1. Alternatively, if the service proposed through the service provision message transmitted in step S631 is not selected by the user, the device (200) can increase the No Pick count of the service provision history information by 1.

[0180] FIG. 8 illustrates an example of data that a voice assistant service provider learns according to one embodiment of the present disclosure.

[0181] As illustrated in FIG. 8, when a user (10) utters a voice command (801), the devices (200) can receive information related to the voice command (801) from the voice assistant service provider (100). In response to the voice command (801), the devices (200) may propose one of the available services and transmit a service provision message containing information related to the proposed service to the voice assistant service provider (100). Additionally, the devices (200) may transmit service provision history information (803) and device information (805) together with the service provision message to the voice assistant service provider (100).

[0182] A voice assistant service providing device (100) can select at least one service providing message from among a plurality of service providing messages and recommend it to the user based on the type of service requested by the voice command (801), service providing history information (803) of each device, and device information (805) of each device.

[0183] FIG. 9 illustrates examples of a service provision message, service provision history information, and device information received by a voice assistant service providing device according to one embodiment of the present disclosure.

[0184] The service provision message list (901) of FIG. 9 illustrates examples of service provision messages received from multiple devices by a voice assistant service providing device (100) according to one embodiment in response to a user's voice command, "I'm hungry, what should I eat?".

[0185] As illustrated in FIG. 9, a service provision message according to one embodiment may include group information, a device identifier, a service type, a service identifier, an application identifier, and result text.

[0186] The group information included in the service provision message may include information regarding whether the device is grouped as being included in a first group associated with the first voice command. For example, the service provision message may include a group identifier representing a first group associated with the first voice command.

[0187] Additionally, the service provision message may include a device identifier representing the device that generates and transmits the service provision message. The service provision message may include at least one of identification information of the service provided by the device in response to the first voice command and the type of the service. For example, the service provision message may include information regarding whether the type of service provided by the device is a notification or an action.

[0188] The type of service included in the service provision message may be the type of service that the voice assistant service providing device (100) determines and transmits as the type of service requested by the first voice command. The voice assistant service providing device (100) may determine the type of service requested by the first voice command and transmit the determined type to the device (200). The device (200) may determine a specific service corresponding to the type of service received from the voice assistant service providing device (100) among various services that the device (200) can provide, and may generate a service provision message related to the specific service and transmit it to the voice assistant service providing device (100).

[0189] The service provision message may include identification information indicating an application that the device uses to provide a service in response to the first voice command. Additionally, the service provision message may include a response message related to a service that the device can provide in response to the first voice command.

[0190] For example, as shown in the first line of the service provision message list (901) of FIG. 9, in response to a user's voice command "I'm hungry, what should I eat?", the TV can generate a service provision message [(I'm hungry / what / shouldI eat) / device 1 / notification / service 1 / application 1 / "There are restaurants featured on TV nearby."] and transmit it to a voice assistant service provision device (100).

[0191] A voice assistant service providing device (100) can see that, based on a service providing message received from a TV, the TV is included in a group associated with a voice command (I'm hungry / what to eat) and intends to provide a notification service, Service 1, through Application 1 in response to the voice command. For example, the TV can suggest a service that recommends restaurants introduced on TV near the user through Application 1, which provides information related to TV programs.

[0192] In addition, as shown in the third line of the service provision message list (901) of FIG. 9, in response to a user's voice command "I'm hungry, what should I eat?", the refrigerator can generate a service provision message [(I'm hungry / what / what to eat) / device 3 / notification / service 3 / application 4 / "The refrigerator is empty."] and transmit it to the voice assistant service provision device (100).

[0193] The voice assistant service providing device (100) can see, based on a service providing message received from the refrigerator, that the refrigerator is included in a group associated with a voice command (I'm hungry / what to eat) and intends to provide a notification service, service 3, through application 4 in response to the voice command. For example, the refrigerator can suggest a service that recommends food that the user can eat or notifies that the refrigerator is empty through application 4, which manages food ingredients stored in the refrigerator.

[0194] The service provision history information list (902) of FIG. 9 illustrates examples of service provision history information received from a plurality of devices in response to a user's voice command, "I'm hungry, what should I eat?" by a voice assistant service providing device (100) according to one embodiment.

[0195] As illustrated in FIG. 9, the service provision history information according to one embodiment may include at least one of whether the device can provide a service that responds to a voice command (isSupported), the number of times a predetermined service proposed by the device for a voice command was selected by the user (Pick count), the number of times a predetermined service proposed by the device for a voice command was not selected by the user (No Pick count), location information of the device, and time information when the predetermined service proposed by the device for a voice command was selected.

[0196] For example, as shown in the service provision history information list (902) of FIG. 9, a TV, a refrigerator, and a smartphone can transmit service provision history information including information that they can provide a service (ei isSupported = yes) in response to a user's voice command "I'm hungry, what should I eat?". On the other hand, an air conditioner can transmit service provision history information including information that it cannot provide a service (ei isSupported = no) in response to a user's voice command "I'm hungry, what should I eat?".

[0197] The voice assistant service providing device (100) can determine whether a service recommended by the voice assistant service providing device (100) to the user has been selected, or whether a service other than the service recommended by the voice assistant service providing device (100) has been selected. The voice assistant service providing device (100) transmits information related to the selected service to the devices, and the devices can learn information related to the selected service. Based on the information related to the selected service, the devices can update the 'number of times the service was selected by the user' or the 'number of times the service was not selected by the user' included in the service provision history information.

[0198] The device information list (903) of FIG. 9 illustrates examples of information about devices constituting each device that a voice assistant service providing device (100) according to one embodiment receives from a plurality of devices.

[0199] Device information according to one embodiment may include information regarding whether the device includes an input / output interface for receiving user input from a user or providing information to a user, and information regarding the performance of each input / output interface. For example, as illustrated in FIG. 9, the device information may include information regarding at least one of the following: whether the device includes a display, the resolution supported by the display, whether the device includes a speaker, the sound quality supported by the speaker, whether the device includes a microphone, the sound quality supported by the microphone, whether the device includes a touchscreen, the sensitivity of the touchscreen, and other information regarding the specifications of the device.

[0200] Other information related to the specifications of a device may include information related to the services that the device can provide. For example, a TV may provide information on TV programs, an air conditioner may provide information related to indoor temperature, humidity, and dust, and a refrigerator may provide information related to food.

[0201] A voice assistant service providing device (100) according to one embodiment may give higher priority to a device that provides a service corresponding to the domain of a user's voice command, based on device specification information included in device information received from devices.

[0202] FIG. 10 illustrates a flowchart of a method in which a voice assistant service providing device assigns priority to a plurality of service providing messages according to one embodiment of the present disclosure.

[0203] A voice assistant service providing device (100) according to one embodiment can select at least one service providing message among a plurality of service providing messages based on at least one of the type of service requested by a voice command, service providing history information of each device, and device information of each device.

[0204] In step S1010, the voice assistant service providing device (100) can identify and group devices included in a group related to voice commands based on isSupported data included in service provision history information provided by a plurality of devices.

[0205] In step S1010, the voice assistant service providing device (100) may select devices that have transmitted information indicating they can provide a service in response to information related to a voice command (e.g., isSupported=Yes) and may assign priority to the selected devices. On the other hand, in step S1013, the voice assistant service providing device (100) may assign the lowest priority to devices that have transmitted information indicating they cannot provide a service in response to information related to a voice command (e.g., isSupported=No).

[0206] In step S1020, the voice assistant service providing device (100) can determine what type of service the device provides based on the service providing message.

[0207] A voice assistant service providing device (100) can determine the type of service requested by a voice command and transmit the determined type to a device (200). The device (200) can determine a specific service corresponding to the type of service received from the voice assistant service providing device (100) among various services that the device (200) can provide, and can generate a service providing message related to the specific service and transmit it to the voice assistant service providing device (100). Accordingly, the type of service included in the service providing message may be the type of service that the voice assistant service providing device (100) determined and transmitted as the type of service requested by the first voice command.

[0208] For example, depending on whether the type of service provided by the selected devices is "notification" or "action execution," the voice assistant service providing device (100) may apply a different method of assigning priority.

[0209] In step S1031, the voice assistant service providing device (100) according to one embodiment may assign priority based on location information and / or time information included in the service provision history information when the type of service requested by the voice command is "notification". For example, the voice assistant service providing device (100) may assign a higher priority to a device as the location of the user and the device are closer, and as the current time is closer to the time when the device was mainly selected. The time interval in which the service provided by each device was selected by the user may be set in various ways, such as in 10-minute intervals, one-hour intervals, or six-hour intervals.

[0210] In step S1033, the voice assistant service providing device (100) according to one embodiment may assign different priorities to devices that were assigned the same priority in step S1031 by further considering the Pick Count (i.e., the number of times a service proposed by the device was selected by the user) included in the service provision history information. For example, the voice assistant service providing device (100) may assign a higher priority to a device that has been selected by the user more often in the past.

[0211] Additionally, in step S1041, the voice assistant service providing device (100) according to one embodiment may assign priority to devices based on device information when the type of service requested by the voice command is "perform operation." The voice assistant service providing device (100) may assign priority based on whether an input / output interface for performing an operation in response to the voice command is provided and the performance of the input / output interface. For example, regarding a voice command requesting to play music, the voice assistant service providing device (100) may assign a higher priority to the device the better the sound quality of the speaker provided in the device.

[0212] In step S1043, the voice assistant service providing device (100) according to one embodiment may assign different priorities to devices assigned the same priority in step S1041 by further considering location information and / or time information included in the service provision history information. For example, if the output sound quality of the speaker is the same, the voice assistant service providing device (100) may assign a higher priority to a device as the location of the user and the device are closer, or as the current time is closer to the time when the device was mainly selected.

[0213] In step S1045, the voice assistant service providing device (100) according to one embodiment may assign different priorities to devices that were assigned the same priority in step S1043 by further considering the Pick Count (i.e., the number of times a service proposed by the device was selected by the user) included in the service provision history information. For example, the voice assistant service providing device (100) may assign a high priority to a device that provides a service that has been frequently selected by the user in the past.

[0214] However, the present disclosure is not limited to the example illustrated in FIG. 10, and the voice assistant service providing device (100) may assign priorities to devices in various orders according to various rules depending on the implementation.

[0215] As an example, a voice assistant service providing device (100) can select one service providing message related to the service most frequently selected by the user (i.e., the service with the highest Pick Count) among a plurality of service providing messages.

[0216] Additionally, the voice assistant service providing device (100) may additionally suggest a service providing message related to another service, the difference between the Pick Count value of the service provided by the selected service providing message and the value provided by the selected service providing message is within a predetermined value (e.g., 1 time).

[0217] For example, referring to the service provision history information list (902) of FIG. 9, the Pick Count for Service 5 proposed by the smartphone is the highest at 5 times, and the Pick Count for Service 1 proposed by the TV is the next highest at 4 times. In this case, the voice assistant service providing device (100) can select the smartphone service providing message that was selected most frequently by the user among the multiple service providing messages and recommend it to the user, and additionally recommend the TV service providing message that was selected next frequently. Based on the selected service providing messages, the voice assistant service providing device (100) can generate and output a response message such as, "Would you like to order pork belly delivery via smartphone? Or, I can recommend a nearby restaurant featured on TV."

[0218] As another example, if the same service providing message with a high Pick Count value is continuously selected during a first period (e.g., one week), the voice assistant service providing device (100) may select and recommend another service providing message with a low Pick Count value based on the user's selection rate. The voice assistant service providing device (100) may also select and recommend services that were not frequently selected in the past to the user, thereby allowing the user to receive various service suggestions.

[0219] As another example, the voice assistant service providing device (100) may exclude from the group associated with the voice command a device that provides a service providing message that has never been selected by the user during a second period (e.g., a quarter). The first period may be a relatively shorter period than the second period. Consequently, the voice assistant service providing device (100) may select and recommend the optimal service preferred by the user to the user.

[0220] Additionally, FIG. 10 illustrates a method in which a voice assistant service providing device (100) sequentially determines information such as whether a device can provide a service that responds to a voice command, category information of the service provided by the device, location information of the device, and information regarding the number of times the service provided by the device has been selected, and assigns priority to the devices. However, the voice assistant service providing device (100) according to various embodiments is not limited to the example illustrated in FIG. 10, and can select at least one service providing message most suitable for a user's voice command by comprehensively considering service providing messages provided by devices, service providing history information, and device information.

[0221] If there are service provision messages with the same priority even after performing the steps illustrated in FIG. 10, the voice assistant service provision device (100) can integrate the service provision messages and propose a service to the user. The voice assistant service provision device (100) can output a response message proposing an integrated service by learning the result text of the service provision messages. As the amount of data that the voice assistant service provision device (100) according to one embodiment learns about services that devices can provide increases, the accuracy of selecting a service provision message suitable for the user can be increased.

[0222] Hereinafter, with reference to FIGS. 11 to 15, examples of specific operation methods of a system providing a voice assistant service according to various embodiments will be described.

[0223] First, FIGS. 11 and 12 illustrate an example of a specific operation method of a system providing a voice assistant service according to a first embodiment of the present disclosure. FIGS. 11 and 12 illustrate a case in which, according to the first embodiment, a voice assistant service providing device (100) is a smart speaker in a home and interacts with a refrigerator (200-1), a smartphone (200-2), a TV (200-3), a wearable device (200-4), and an exercise machine (200-5).

[0224] As illustrated in FIG. 11, the voice assistant service provider (100) can receive a user's voice command, "I'm hungry, what should I eat?"

[0225] The voice assistant service providing device (100) can determine which of the plurality of candidate devices (200-1, 200-2, 200-3, 200-4, 200-5) connected to the voice assistant service providing device (100) will transmit information related to a user's voice command.

[0226] The voice assistant service providing device (100) can convert a voice command into text and interpret the converted text. The voice assistant service providing device (100) can determine a plurality of devices (200-1, 200-2, 200-3, 200-4) related to the interpreted text among a plurality of candidate devices (200-1, 200-2, 200-3, 200-4, 200-5) using a device determination model.

[0227] According to one embodiment, a voice assistant service providing device (100) analyzes text using a natural language understanding model and, based on the analysis result of the text, can determine among a plurality of candidate devices (200-1, 200-2, 200-3, 200-4, 200-5) a plurality of devices (200-1, 200-2, 200-3, 200-4) that can provide a service according to the user's intention (e.g., menu recommendation, restaurant decision, food order, etc.).

[0228] According to one embodiment, a voice assistant service providing device (100) can determine, through a device determination model, that among a plurality of candidate devices (200-1, 200-2, 200-3, 200-4, 200-5), devices (200-1, 200-2, 200-3, 200-4) for which the probability of providing a service according to the user's intention is greater than a threshold value are all devices that will transmit information related to voice commands. The voice assistant service providing device (100) can determine that, in the case of an exercise machine (200-5), the relevance to a service according to the user's intention (e.g., menu recommendation, restaurant selection, food ordering, etc.) is lower than that of other devices. Accordingly, the voice assistant service providing device (100) may decide to transmit information related to the user's voice command only to the refrigerator (200-1), smartphone (200-2), TV (200-3), and wearable device (200-4) among a plurality of candidate devices (200-1, 200-2, 200-3, 200-4, 200-5).

[0229] As illustrated in FIG. 11, in one embodiment, a voice assistant service provider (100) can transmit information related to voice commands to only selected devices, rather than transmitting information related to voice commands to all connected devices, even if the number of connected devices increases. Therefore, there is an advantage in that the voice assistant service provider (100) can prevent the waste of communication resources that may occur while communicating with many devices.

[0230] A voice assistant service providing device (100) transmits information related to a voice command to a plurality of devices (200-1, 200-2, 200-3, 200-4). For example, the voice assistant service providing device (100) can transmit to the plurality of devices (200-1, 200-2, 200-3, 200-4) information indicating that the type of service requested by the voice command is "Notification," and that [I'm hungry / what / should I eat] is the result of interpreting the text converted from the voice command.

[0231] Each of the plurality of devices (200-1, 200-2, 200-3, 200-4) can determine whether each device can provide a service in response to information related to a received voice command. For example, each device can select services among the services that the device can provide that the type of service is "notification" and determine whether there is a service among the selected services that corresponds to the voice command.

[0232] Referring to FIG. 11, the refrigerator (200-1), smartphone (200-2), and TV (200-3) determine that they can provide a service in response to a user's voice command, and the wearable device (200-4) determines that it cannot provide a service in response to a user's voice command. If it is determined that a service can be provided in response to information related to a voice command, the device may generate a service provision message to be transmitted to a voice assistant service provision device (100). The service provision message transmitted to the voice assistant service provision device (100) may be generated according to the format exemplified in FIG. 9.

[0233] First, in response to a user's voice command, "I'm hungry, what should I eat?", the refrigerator (200-1) can generate a service provision message [(I'm hungry / what / to eat) / refrigerator / notification / service 3 / application 4 / "The refrigerator is empty."] and transmit it to the voice assistant service provider device (100). The refrigerator (200-1) can transmit a service provision message to the voice assistant service provider device (100) that includes information that the refrigerator (200-1) is included in the group associated with the voice command (I'm hungry / what / to eat) and intends to provide a notification service, Service 3, through Application 4 in response to the voice command. For example, as illustrated in FIG. 11, the refrigerator (200-1) can propose a service that notifies that the refrigerator (200-1) is empty through Application 4, which manages the food ingredients stored inside the refrigerator (200-1).

[0234] Next, in response to a user's voice command, "I'm hungry, what should I eat?", the smartphone (200-2) can generate a service provision message [(I'm hungry / what / to eat) / device 4 / notification / service 5 / application 3 / "Shall I order pork belly delivery?"] and send it to the voice assistant service provider device (100). The smartphone (200-2) can send a service provision message to the voice assistant service provider device (100) that includes information that the smartphone (200-2) is included in the group associated with the voice command (I'm hungry / what / to eat) and intends to provide a notification service, Service 5, through Application 3 in response to the voice command. For example, as illustrated in FIG. 11, the smartphone (200-2) can suggest a service to order pork belly delivery, which is the user's most preferred food, through Application 3, which orders food delivery.

[0235] In response to a user's voice command, "I'm hungry, what should I eat?", the TV (200-3) can generate a service provision message [(I'm hungry / what / should / eat) / device 1 / notification / service 1 / application 1 / "There are restaurants featured on TV nearby."] and transmit it to a voice assistant service provider device (100). The TV (200-3) can transmit a service provision message to the voice assistant service provider device (100) that includes information that the TV (200-3) is included in the group associated with the voice command (I'm hungry / what / should / eat) and intends to provide a notification service, Service 1, through Application 1 in response to the voice command. For example, as illustrated in FIG. 11, the TV (200-3) can propose a service that recommends restaurants featured on TV near the user through Application 1, which provides TV program information.

[0236] Meanwhile, although not illustrated in FIG. 11, a refrigerator (200-1), a smartphone (200-2), and a TV (200-3) can transmit service provision history information along with a service provision message to a voice assistant service providing device (100). The service provision history information transmitted to the voice assistant service providing device (100) can be generated according to the format exemplified in FIG. 9.

[0237] For example, the refrigerator (200-1) can provide a service that responds to a user's voice command, "I'm hungry, what should I eat?" (isSupported=yes), and can transmit service provision history information to a voice assistant service providing device (100), such that the service has been selected by the user once (Pick Count=1), has been suggested but not selected eight times (No Pick Count=1), is located in the kitchen, and the time when the provided service is mainly selected is 18:00.

[0238] A smartphone (200-2) can provide a service that responds to a user's voice command, "I'm hungry, what should I eat?" (isSupported=yes), and can transmit service provision history information to a voice assistant service providing device (100), such that the number of times the service was selected by the user is 5 (Pick Count=5), the number of times it was suggested but not selected is 4 (No Pick Count=4), it is located in the living room, and the time when the provided service is mainly selected is 22:30.

[0239] TV (200-3) can provide a service that responds to a user's voice command "I'm hungry, what should I eat?" (isSupported=yes), and can transmit service provision history information to a voice assistant service provider device (100) that the number of times the service was selected by the user is 4 (Pick Count=4), the number of times it was suggested but not selected is 6 (No Pick Count=6), it is located in the living room, and the time when the provided service is mainly selected is 22:30.

[0240] Next, referring to FIG. 12, the voice assistant service providing device (100) can group devices based on service providing messages received from a plurality of devices (200-1, 200-2, 200-3, 200-4). The voice assistant service providing device (100) can determine that a refrigerator (200-1), a smartphone (200-2), and a TV (200-3) that sent a service providing message indicating that they are included in a group associated with a voice command (I'm hungry / what to eat) are included in one group.

[0241] The voice assistant service providing device (100) can transmit group information to the refrigerator (200-1), smartphone (200-2), and TV (200-3) that the refrigerator (200-1), smartphone (200-2), and TV (200-3) are included in one group.

[0242] The voice assistant service providing device (100) can select at least one service providing message from among the service providing messages of the grouped refrigerator (200-1), smartphone (200-2), and TV (200-3). The voice assistant service providing device (100) can select at least one service providing message from among the grouped service providing messages based on the service providing history information of each device received along with the service providing message.

[0243] A voice assistant service providing device (100) according to one embodiment may assign priority based on location information and / or time information included in service provision history information when the type of service requested by a voice command is "notification". For example, if the current time is 22:30, the voice assistant service providing device (100) may assign the same high priority to a smartphone (200-2) and a TV (200-3) for which the time the device was mainly selected was 22:30.

[0244] Next, the voice assistant service provider (100) can select a service provider message from the smartphone (200-2) with a higher Pick Count among the smartphone (200-2) and the TV (200-3).

[0245] The voice assistant service providing device (100) can output a response message that responds to a voice command based on a selected service providing message. As shown in FIG. 12, the voice assistant service providing device (100) can generate and output a response message saying "Shall we order pork belly delivery?" based on a selected service providing message.

[0246] In FIG. 12, a voice assistant service providing device (100) selects one service providing message with the highest priority and generates and outputs a response message based on the selected service providing message. However, voice assistant service providing device (100) according to various embodiments of the present disclosure is not limited to the example shown in FIG. 12.

[0247] As illustrated in FIG. 13, a voice assistant service providing device (100) according to one embodiment can select a plurality of service providing messages and generate and output a response message that includes information about the plurality of service providing messages.

[0248] FIG. 13 illustrates an example of a specific operation method of a system providing a voice assistant service according to a second embodiment of the present disclosure.

[0249] As illustrated in FIG. 13, the voice assistant service providing device (100) can receive a user's voice command, "Shall we do some exercise today?" The voice assistant service providing device (100) can determine a plurality of devices (200-1, 200-2, 200-3, 200-4, 200-5) among a plurality of connected candidate devices to transmit information related to the voice command.

[0250] The voice assistant service providing device (100) transmits information related to voice commands to a plurality of devices (200-1, 200-2, 200-3, 200-4, 200-5). For example, the voice assistant service providing device (100) can transmit to the plurality of devices (200-1, 200-2, 200-3, 200-4, 200-5) information indicating that the type of service requested by the voice command is "Notification" and that the result of interpreting the text converted from the voice command is "[Shall we try exercising today]".

[0251] Each of the plurality of devices (200-1, 200-2, 200-3, 200-4, 200-5) can determine whether each device can provide a service in response to information related to a received voice command. Referring to FIG. 13, the TV (200-3), the wearable device (200-4), and the exercise equipment (200-5) determine that they can provide a service in response to the user's voice command, while the refrigerator (200-1) and the smartphone (200-2) determine that they cannot provide a service in response to the user's voice command. If it is determined that a device can provide a service in response to information related to a voice command, the device can generate a service provision message to be transmitted to the voice assistant service provision device (100). The service provision message transmitted to the voice assistant service provision device (100) can be generated according to the format exemplified in FIG. 9.

[0252] First, in response to a user's voice command, "Shall we do some exercise today?", the TV (200-3) may suggest a service recommending that the TV channel be changed to a sports channel, even though it has not been notified that it is included in the group associated with the voice command (shall we / do / exercise today). The TV (200-3) may generate a service provision message containing information, "I will play the sports channel for you," and send it to the voice assistant service provision device (100).

[0253] Next, in response to a user's voice command "Shall we do some exercise today?", the wearable device (200-4) can send a service provision message that includes information that the wearable device (200-4) is included in the group associated with the voice command (shall we / do / exercise today), and suggests a service that recommends aerobic exercise through an application that records the user's exercise history and recommends suitable exercise.

[0254] The wearable device (200-4) can generate a service provision message containing information such as "Since you did strength training yesterday, I think it would be good to do aerobic exercise today" and send it to the voice assistant service provision device (100).

[0255] Next, in response to a user's voice command, "Shall we do some exercise today?", the exercise equipment (200-5) can send a service provision message that includes information that the exercise equipment (200-5) is included in the group associated with the voice command (shall we / do / exercise today), and suggests a service that sets the user's preferred exercise time on a timer through an application that records the user's exercise history. The exercise equipment (200-5) can generate a service provision message that includes information that "I will set the exercise time to 30 minutes" and send it to the voice assistant service provision device (100).

[0256] Additionally, although not illustrated in FIG. 13, the TV (200-3), wearable device (200-4), and exercise equipment (200-5) may transmit service provision history information and device information along with a service provision message to the voice assistant service providing device (100). The service provision history information and device information transmitted to the voice assistant service providing device (100) may be generated according to the format exemplified in FIG. 9.

[0257] The voice assistant service providing device (100) can group devices based on service providing messages received from a plurality of devices (200-1, 200-2, 200-3, 200-4, 200-5). The voice assistant service providing device (100) can determine that a wearable device (200-4) and an exercise device (200-5) that sent a service providing message indicating that they are included in a group associated with the voice command (shall we try exercising today?) are included in one group.

[0258] In FIG. 13, the voice assistant service providing device (100) identifies information from the service providing message regarding whether a device is included in a group related to a voice command, and determines whether each device is included in a group related to a voice command based on the identified information. That is, the voice assistant service providing device (100) determines whether each device is included in a group related to a voice command based on group information that the device has learned and stored in advance. Therefore, the response speed can be increased because the voice assistant service providing device (100) can omit the operation of analyzing service providing messages and service providing history information received from the devices and grouping the devices.

[0259] Subsequently, the voice assistant service provider (100) can analyze service provision messages received from the devices to regroup the devices and update group information. For example, the voice assistant service provider (100) can update group information to include not only the wearable device (200-4) and exercise equipment (200-5) but also the TV (200-3) in one group. The voice assistant service provider (100) can transmit the updated group information to the TV (200-3), the wearable device (200-4), and the exercise equipment (200-5).

[0260] A voice assistant service providing device (100) can select at least one service providing message from among service providing messages of a wearable device (200-4) and an exercise device (200-5) determined to be included in one group, based on at least one of the service providing history information and device information of each device received together with the service providing message. In FIG. 13, the voice assistant service providing device (100) can select all service providing messages of the wearable device (200-4) and the exercise device (200-5).

[0261] The voice assistant service providing device (100) can output a response message that responds to a voice command based on selected service providing messages. As illustrated in FIG. 13, the voice assistant service providing device (100) can combine selected service providing messages to generate and output a single response message. The voice assistant service providing device (100) can combine service providing messages from a wearable device (200-4) and an exercise machine (200-5) to generate and output a response message saying, “Since you did strength training yesterday, how about doing 30 minutes of aerobic exercise using the exercise machine?”

[0262] Meanwhile, a voice assistant service providing device (100) according to one embodiment can identify devices included in one group by considering service provision history information or device information of each device received together with a service provision message. FIG. 14 illustrates an example of a specific operation method of a system providing a voice assistant service according to a third embodiment of the present disclosure.

[0263] As illustrated in FIG. 14, a voice assistant service providing device (100) according to one embodiment can receive a user's voice command, "Play a song." The voice assistant service providing device (100) can determine a plurality of devices (200-1, 200-2, 200-3, 200-4, 200-5) among a plurality of connected candidate devices to transmit information related to the voice command.

[0264] The voice assistant service providing device (100) transmits information related to voice commands to a plurality of devices (200-1, 200-2, 200-3, 200-4, 200-5). For example, the voice assistant service providing device (100) can transmit to the plurality of devices (200-1, 200-2, 200-3, 200-4, 200-5) information indicating that the type of service requested by the voice command is “perform action” and that the result of interpreting the text converted from the voice command is [play song / play].

[0265] Each of the plurality of devices (200-1, 200-2, 200-3, 200-4, 200-5) can determine whether to provide a service in response to information related to a received voice command. The plurality of devices (200-1, 200-2, 200-3, 200-4, 200-5) can determine whether to provide a service based on the presence or absence of a speaker for playing a song.

[0266] Referring to FIG. 14, a refrigerator (200-1), a smartphone (200-2), a TV (200-3), and a wearable device (200-4) determine that they can provide a service in response to a user's voice command, and an exercise machine (200-5) that does not have a speaker determines that it cannot provide a service in response to a user's voice command. If it is determined that a service can be provided in response to information related to a voice command, the device may generate a service provision message to be transmitted to a voice assistant service provision device (100). The service provision message transmitted to the voice assistant service provision device (100) may be generated according to the format exemplified in FIG. 9.

[0267] In response to a voice command "Play a song," the refrigerator (200-1) may send a service providing message that includes information that it is included in a group associated with the voice command (play song / song), and suggests a service to play a song recently played by the user. The refrigerator (200-1) may generate a service providing message that includes information that it "plays a recently listened TWICE song" and send it to a voice assistant service providing device (100).

[0268] Next, in response to a voice command "Play a song," the smartphone (200-2), TV (200-3), and wearable device (200-4) may transmit a service providing message that suggests a service to play a song recently played by the user, including information that the devices are included in the group associated with the voice command (play song). The smartphone (200-2), TV (200-3), and wearable device (200-4) may generate a service providing message that includes information "Playing a recently listened TWICE song" and transmit it to the voice assistant service providing device (100).

[0269] Additionally, as illustrated in FIG. 14, a refrigerator (200-1), a smartphone (200-2), a TV (200-3), and a wearable device (200-4) can transmit service provision history information and device information along with a service provision message to a voice assistant service providing device (100). The service provision history information and device information transmitted to the voice assistant service providing device (100) can be generated according to the format exemplified in FIG. 9.

[0270] For example, the refrigerator (200-1) may further transmit a service provision history including information that the refrigerator (200-1) is located in the kitchen. The smartphone (200-2), TV (200-3), and wearable device (200-4) may further transmit a service provision history including information that the devices are located in the living room.

[0271] The voice assistant service providing device (100) can group devices based on service providing messages received from a plurality of devices (200-1, 200-2, 200-3, 200-4, 200-5). The voice assistant service providing device (100) can determine that a refrigerator (200-1), a smartphone (200-2), a TV (200-3), and a wearable device (200-4) that sent a service providing message indicating that they are included in a group associated with a voice command (play a song) are included in one group.

[0272] A voice assistant service providing device (100) can select at least one service providing message from among service providing messages of a refrigerator (200-1), a smartphone (200-2), a TV (200-3), and a wearable device (200-4) that are determined to be included in one group, based on at least one of the service providing history information and device information of each device received together with the service providing message.

[0273] First, the voice assistant service providing device (100) can identify a smartphone (200-2), a TV (200-3), and a wearable device (200-4) located in a "living room" close to the voice assistant service providing device (100) or the user's current location, based on the location information of the device included in the service provision history information.

[0274] Next, the voice assistant service provider (100) can prioritize the device based on the device information and the device's speaker performance. The voice assistant service provider (100) can select the service provision message of the TV (200-3) with the best speaker performance.

[0275] The voice assistant service providing device (100) can generate and output a response message, "Playing a recently listened TWICE song on the TV," based on a selected service providing message.

[0276] Meanwhile, FIG. 15 illustrates an example of a specific operation method of a system providing a voice assistant service according to the fourth embodiment of the present disclosure.

[0277] As illustrated in FIG. 15, a voice assistant service providing device (100) according to one embodiment can receive a user's voice command "I'm depressed."

[0278] The voice assistant service providing device (100) transmits information related to voice commands to a plurality of devices (200-1, 200-2, 200-3, 200-4, 200-5). For example, the voice assistant service providing device (100) can transmit to the plurality of devices (200-1, 200-2, 200-3, 200-4, 200-5) information indicating that the result of interpreting the text converted from the voice command is [I'm depressed] and that the type of service requested by the voice command is "Notification".

[0279] Each of the plurality of devices (200-1, 200-2, 200-3, 200-4, 200-5) can determine whether each device can provide a service in response to information related to a received voice command. Referring to FIG. 15, the refrigerator (200-1), smartphone (200-2), and TV (200-3) determine that they can provide a service in response to the user's voice command, while the wearable device (200-4) and exercise equipment (200-5) determine that they cannot provide a service in response to the user's voice command. If it is determined that a device can provide a service in response to information related to a voice command, the device can generate a service provision message to be transmitted to the voice assistant service provision device (100). The service provision message transmitted to the voice assistant service provision device (100) can be generated according to the format exemplified in FIG. 9.

[0280] First, in response to a user's voice command "I'm depressed," the refrigerator (200-1) can send a service provision message including information that the refrigerator (200-1) is included in a group associated with the voice command "I'm depressed," and service information that recommends ice cream to the user through an application that manages food ingredients stored in the refrigerator (200-1).

[0281] Next, in response to a user's voice command "I'm depressed," the smartphone (200-2) can transmit a service provision message including information that the smartphone (200-2) is included in a group associated with the voice command "I'm depressed" and service information recommending music playback through a music playback application.

[0282] Next, in response to a user's voice command "I'm depressed," the TV (200-3) can send a service provision message containing information that the TV (200-3) is included in the group associated with the voice command "I'm depressed" and information recommending a comedy program through an application that provides TV program information.

[0283] The voice assistant service providing device (100) can group devices based on service providing messages received from a plurality of devices (200-1, 200-2, 200-3, 200-4, 200-5). The voice assistant service providing device (100) can determine that a refrigerator (200-1), a smartphone (200-2), and a TV (200-3) that sent a service providing message indicating that they are included in a group associated with a voice command (I'm depressed) are included in one group.

[0284] The voice assistant service providing device (100) can select at least one service providing message from among the service providing messages of the grouped refrigerator (200-1), smartphone (200-2), and TV (200-3). The voice assistant service providing device (100) can select at least one service providing message from among the grouped service providing messages based on the service providing history information of each device received along with the service providing message.

[0285] A voice assistant service providing device (100) according to one embodiment may assign priority based on location information and / or time information included in service provision history information when the type of service requested by a voice command is "notification". For example, the highest priority may be assigned to a smartphone (200-2) where the time at which the device was mainly selected is close to the current time.

[0286] The voice assistant service providing device (100) can output a response message that responds to a voice command based on a selected service providing message. As illustrated in FIG. 15, the voice assistant service providing device (100) can generate and output a response message saying "I recommend music that makes you feel good" based on a selected service providing message.

[0287] Hereinafter, the configuration of a voice assistant service providing device (100) according to one embodiment is described. Each configuration of the voice assistant service providing device (100) described below can perform each step of the method of operation of the voice assistant service providing device (100) described above. Therefore, descriptions that overlap with the description above are omitted.

[0288] FIG. 16 is a block diagram of a voice assistant service providing device (100) according to one embodiment.

[0289] A voice assistant service providing device (100) according to one embodiment of the present disclosure may be a fixed terminal or a mobile terminal implemented as a computer device. The voice assistant service providing device (100) may be, for example, at least one of a smartphone, a mobile phone, a navigation system, a computer, a laptop, a digital broadcasting terminal, an artificial intelligence speaker, a PDA (Personal Digital Assistants), a PMP (Portable Multimedia Player), and a tablet PC, but is not limited thereto. The voice assistant service providing device (100) may communicate with other devices and / or servers via a network using a wireless or wired communication method.

[0290] Referring to FIG. 16, a voice assistant service provider (100) may include a receiver (110), a processor (120), a communication unit (130), a memory (140), and an output unit (150). Not all components shown in FIG. 16 are essential components of the voice assistant service provider (100). The voice assistant service provider (100) may be implemented by more components than those shown in FIG. 16, or by fewer components than those shown in FIG. 16. For example, as shown in FIG. 18, a voice assistant service provider (100) according to some embodiments may further include a user input unit (2100), a sensing unit (2400), and an A / V input unit (2600). At least one component shown in FIG. 16 through 18 may perform the operation of FIG. 4 and FIG. 5. Therefore, descriptions that overlap with the descriptions above with reference to FIGS. 4 and FIGS. 5 are omitted.

[0291] A receiver (110) according to one embodiment of the present disclosure can receive voice commands from a user. For example, the receiver (110) can receive voice commands by converting external sound into electrical acoustic data using a microphone. Although FIG. 16 shows the receiver (110) included inside a voice assistant service provider (100), a receiver (110) according to another embodiment may be implemented in a form that is included in a separate device and connected to the voice assistant service provider (100) via wired or wireless connection.

[0292] A memory (140) according to one embodiment of the present disclosure may store instructions for providing voice assistant services, various models used for voice recognition, neural networks, or dictionary information.

[0293] A processor (120) according to one embodiment of the present disclosure can provide a voice assistant service according to various embodiments of the present disclosure by controlling a receiving unit (110), a communication unit (130), and an output unit (150) by executing one or more instructions stored in a memory (140). Although FIG. 16 illustrates a voice assistant service providing device (100) comprising one processor (120), the present disclosure is not limited to the illustrated embodiment. The voice assistant service providing device (100) may include a plurality of processors. When the voice assistant service providing device (100) includes a plurality of processors, the operation and function of the processor (120) described below may be partially performed in the plurality of processes.

[0294] A processor (120) according to one embodiment can receive a first voice command from a user through a receiver (110).

[0295] The processor (120) can convert a first voice command into a first text, interpret the converted first text, and transmit information regarding the result of interpreting the first text and the type of service requested by the first voice command as information related to the first voice command.

[0296] The processor (120) can determine among a plurality of candidate devices which devices will transmit information related to the first voice command. The processor (120) can determine the user's intent based on the result of interpreting the first text. The processor (120) can determine among a plurality of candidate devices which devices will transmit the first voice command based on the relationship between the user's intent and the plurality of candidate devices. The processor (120) can determine among a plurality of candidate devices which devices have a high probability of providing a service according to the user's intent through a device determination model.

[0297] The processor (120) can control the communication unit (130) to transmit information related to a first voice command received from a user to a plurality of devices based on a judgment result. The communication unit (130) according to one embodiment can communicate with an external device, device, or server via wired communication or wireless communication. The communication unit (130) may include a short-range communication module, a wired communication module, a mobile communication module, a broadcast reception module, etc.

[0298] Multiple devices may generate and transmit a service provision message in response to information related to a user's first voice command received from a voice assistant service providing device (100). Multiple devices may further transmit at least one of service provision history information and device information to the voice assistant service providing device (100).

[0299] The processor (120) can receive a plurality of service provision messages responding to a first voice command and service provision history information of the plurality of devices from each of the plurality of devices through the communication unit (130). Additionally, the processor (120) can further receive device information constituting each device from the plurality of devices through the communication unit (130).

[0300] A service provision message received from a device may include at least one of whether the device can provide a service in response to a voice command, whether the device is included in a group related to the voice command, identification information of the device, identification information of the service provided by the device in response to the voice command, the type of service, and identification information of an application used to provide the service.

[0301] Service provision history information refers to information regarding past history related to services provided or proposed by the device in response to information related to a user's predetermined voice command received from the voice assistant service providing device (100). The service provision history information may include at least one of whether the device can provide a service that responds to a voice command, the number of times a service proposed by the device in response to a voice command was selected by the user, and situational information when a service proposed by the device in response to a voice command was selected.

[0302] A processor (120) according to one embodiment can select at least one of a plurality of service provision messages based on service provision history information.

[0303] A processor (120) according to one embodiment can perform grouping to identify devices included in a first group related to a first voice command among a plurality of devices. The processor (120) can group devices based on at least one of service provision messages and service provision history information received from the devices, and learn the grouped results.

[0304] Additionally, the processor (120) may select at least one of the service provision messages received from devices identified as being included in the first group based on service provision history information. The processor (120) may select at least one of the multiple service provision messages by further considering the device information of the multiple devices.

[0305] A processor (120) according to one embodiment may use a service recommendation model stored in memory (140) to select at least one of a plurality of service provision messages. The service recommendation model may be an artificial intelligence algorithm learned by at least one of a predetermined voice command, a service provision history of a plurality of devices for the predetermined voice command, and device information of each of the plurality of devices.

[0306] The processor (120) may assign weights or priorities to a plurality of service provision messages based on service provision history information and device information of the devices. The processor (120) may assign priorities or weights to a plurality of service provision messages based on at least one of whether each device provides a service in response to a first voice command, the type of service provided by each device in response to the first voice command, the number of times the service provided by each device was selected for the first voice command, situation information when the service provided by each device was selected for the first voice command, and device information constituting each device. The processor (120) may select at least one of the plurality of service provision messages based on the assigned weights or priorities.

[0307] A processor (120) according to one embodiment can generate a response message that responds to a first voice command based on at least one selected service provision message and output it through an output unit (150).

[0308] The processor (120) may generate a response message that includes service information related to at least one selected service provision message. An output unit (150) according to one embodiment may include a speaker capable of outputting a response message or a display capable of outputting a video signal.

[0309] The user can listen to a response message responding to a first voice command and decide whether to accept a service proposed by the voice assistant service provider (100). Based on the decision, the user can utter a second voice command.

[0310] A processor (120) according to one embodiment can receive a second voice command from a user through a receiver (110).

[0311] A processor (120) according to one embodiment can determine a service selected by a user among services provided by a plurality of devices based on a voice command received from a user, a predetermined action of the user, or user input. The processor (120) can transmit information related to the service selected by the user to a plurality of devices through a communication unit (130). Additionally, the processor (120) can learn a service recommendation model stored in a memory (140) based on information related to the service selected by the user. The voice assistant service providing device (100) can continuously learn a service recommendation model by the user's voice command in order to suggest an optimal service preferred by the user among various services provided by a plurality of devices.

[0312] A processor (120) of a voice assistant service providing device (100) according to one embodiment may operate to generate a response message more simply when a specific device is specified in a user's first voice command, a request for a service of a specific device is specified, or information related to a specific device is included. The processor (120) may omit the operation of determining a plurality of devices and the operation of selecting at least one of the service providing messages received from the plurality of devices. A processor (120) according to one embodiment may transmit information related to the first voice command to a specific device, receive a service providing message from a specific device, and generate and output a response message that includes information related to the service within the received service providing message.

[0313] Meanwhile, the processor (120) of the voice assistant service providing device (100) according to one embodiment can obtain device information from a plurality of devices connected to the voice assistant service providing device (100). For example, the device information may include resource information and / or capability information of the corresponding device. Based on the device information of the plurality of devices, the processor (120) can play a more leading role in providing voice assistant services. The processor (120) can adaptively select a plurality of devices or select at least one service providing message according to the resource information and / or capability information of the plurality of devices.

[0314] Below, we will examine the operation of each component of the voice assistant service providing device (100) in more detail with reference to FIG. 17.

[0315] FIG. 17 illustrates a block diagram of a voice assistant service providing device (100) according to one embodiment of the present disclosure.

[0316] A voice assistant service providing device (100) according to one embodiment can receive a first voice command from a user through a microphone (110). The microphone (110) can convert the sound received from the user into a first voice command, which is an electrical signal.

[0317] A processor (120) of a voice assistant service providing device (100) according to one embodiment can extract information related to the first voice command from the first voice command and transmit information related to the first voice command to a plurality of devices through a communication unit (130).

[0318] Specifically, the processor (120) may include a speech recognition unit (121) that performs speech recognition. In one embodiment, the speech recognition unit (121) may convert a first speech command into a first text through an Auto Speech Recognition (ASR) module (122) and interpret the first text through a Natural Language Understanding module (123). The speech recognition unit (121) may convert the first speech command into a first text and interpret the first text using an artificial intelligence neural network (141) designed for speech recognition.

[0319] The device determination unit (129) of the processor (120) can determine, using a device determination model (142), a plurality of candidate devices connected to the voice assistant service providing device (100) to transmit information related to the user's first voice command. The device determination model (142) may include a matching model that determines the relationship between the user's intention obtained from the first text interpretation result and the plurality of candidate devices. The device determination unit (129) can determine a predetermined number of multiple devices among the plurality of candidate devices that have a high probability of providing a service according to the user's intention through the device determination model (142).

[0320] A processor (120) of a voice assistant service providing device (100) according to one embodiment can transmit information related to a voice command output from a voice recognition unit (121) to a plurality of devices through a communication unit (130).

[0321] “Information related to a voice command” transmitted to multiple devices may include the result of processing the voice command. The result of processing the voice command may include at least one of a feature vector extracted from the user’s voice command, text converted from the user’s voice command, a result output from the text after undergoing Natural Language Understanding (NLU) processing, a type of service requested by the voice command determined based on the result of interpreting the text, the volume of the user’s voice command, and the SNR of the user’s voice command.

[0322] A processor (120) of a voice assistant service providing device (100) according to one embodiment can transmit information regarding the result of interpreting a first text and the type of service requested by the first voice command to a plurality of devices. A voice recognition unit (121) can extract text data indicating the user's speech intent according to a predetermined rule (e.g., the 5 Ws and 1 H) through natural language processing of the first text converted from the first voice command. The processor (120) can transmit the text data extracted by the voice recognition unit (121) to a plurality of devices as the result of interpreting the first text. Additionally, the voice recognition unit (121) can determine whether the type of service requested by the first voice command is a notification or an action based on the result of interpreting the first text.

[0323] The voice recognition unit (121) can determine that the type of service requested by the first voice command is "operation execution" when it is determined that the first voice command requests the device to perform a predetermined operation (e.g., media playback, recording, power on / off, etc.), and otherwise determine that the type of service requested by the voice command is "operation execution".

[0324] For example, when a voice command “I’m hungry, what should I eat?” is received from a user, the voice assistant service providing device (100) can transmit to multiple devices the type of service requested by the voice command (notification of information) along with text data (I’m hungry / what / should I eat) extracted from the voice command.

[0325] A plurality of devices (200-1, 200-2, 200-3) connected to a voice assistant service providing device (100) can determine whether each device can provide a service that responds to the first voice command based on information related to the received first voice command. For example, the plurality of devices (200-1, 200-2, 200-3) can receive from the voice assistant service providing device (100) the result of interpreting the first text in which the first voice command is converted, and determine whether the service can be provided based on the received text interpretation result.

[0326] Each device can identify services that the device (200) can provide based on at least one of information about devices constituting the device (200) (e.g., screen, input interface, etc.), information about software operating on the device (200) (e.g., application, etc.), and general situational information (e.g., time information, and location information, etc.) that is stored in advance in the device (200). Each device can determine whether among the services identified as being available to the device, there is a service that can be a response to the first voice command.

[0327] Each device may determine whether, among the services identified as being available to the device, a service corresponding to the type of service requested by the first voice command is included. If, among the identified services, at least one service corresponding to the type of service requested by the first voice command is included, the device may determine whether, among the at least one service, a service that can be a response to the first voice command is included.

[0328] If, among the services identified as being available to the device, a service that can be a response to the first voice command is included, the device may determine that it can provide the service in response to information related to the received first voice command.

[0329] If the device determines that it can provide a service in response to information related to the first voice command, the device may generate a service provision message and transmit it to the voice assistant service provision device (100). The device (200) may generate a service provision message by selecting one service corresponding to the type of service requested by the first voice command from among the services identified as being available to the device.

[0330] "Service provision message" refers to a message output by a device in response to information related to a user's voice command received from a voice assistant service providing device (100). For example, the service provision message may include at least one of whether the device can provide a service that responds to a voice command, whether the device is included in a group related to the voice command, identification information of the device, identification information of a service provided by the device in response to a voice command, a type of service, and identification information of an application used to provide the service.

[0331] A voice assistant service providing device (100) according to one embodiment can receive a service providing message from each of a plurality of devices (200-1, 200-2, 200-3). Additionally, the voice assistant service providing device (100) can further receive at least one of service providing history information and device information from each of the plurality of devices (200-1, 200-2, 200-3).

[0332] “Service provision history information” refers to information regarding past history related to services provided or proposed by the device in response to information related to a user’s predetermined voice command received from the voice assistant service providing device (100). The service provision history information may include at least one of whether the device provides a service for the first voice command, the number of times the service proposed by the device for the first voice command was selected by the user, and situation information when the service was selected for the first voice command (e.g., time information, location information of the user or device, etc.).

[0333] “Device information” means information about the devices constituting the device. Specifically, the device information may include information regarding whether the device includes an input / output interface for receiving user input from a user or providing information to a user, and information related to the performance of each input / output interface. For example, the device information may include information regarding whether the device includes a display, the resolution supported by the display, whether the device includes a microphone, the sound quality supported by the microphone, whether the device includes a speaker, the sound quality supported by the speaker, and at least one of the fields of services that the device can provide.

[0334] A voice assistant service providing device (100) according to one embodiment can recommend a service suitable for a user among services that can be provided from a plurality of devices. The processor (120) of the voice assistant service providing device (100) may include a service recommendation unit (124) for recommending a service suitable for a user based on service provision messages received from a plurality of devices.

[0335] The group identification unit (125) of the service recommendation unit (124) can group devices based on at least one of service provision messages and service provision history information received from multiple devices.

[0336] In the case where multiple devices (200-1, 200-2, 200-3) have not learned group information related to the first voice command, the group identification unit (125) according to one embodiment may analyze service provision history information received from the multiple devices (200-1, 200-2, 200-3) and determine the devices included in the first group corresponding to the user's first voice command based on the analysis result. The group identification unit (125) may group devices capable of providing a service that responds to the first voice command into one group.

[0337] The group identification unit (125) can obtain information (isSupported data) regarding whether a device can provide a service that responds to a first voice command from the service provision history information. Based on the service provision history information received from a plurality of devices, the group identification unit (125) can identify devices included in a first group corresponding to the user's first voice command among the plurality of devices. The group identification unit (125) can output service provision messages received from the identified first group of devices (hereinafter referred to as 'service provision messages of the first group').

[0338] When learning has been performed on information grouping devices related to the first voice command, the group identification unit (125) according to one embodiment can identify devices included in the first group based on group information included in service provision messages received from a plurality of devices (200-1, 200-2, 200-3). The group identification unit (125) can obtain information regarding whether a device is included in the group related to the first voice command from the service provision message. The group identification unit (125) can output service provision messages of the first group identified based on the service provision message.

[0339] Next, the service selection unit (126) of the service recommendation unit (124) can select at least one service provision message from among the service provision messages of the first group.

[0340] According to one embodiment, the service selection unit (126) can select all service provision messages received from the devices of the first group.

[0341] For example, the group identification unit (125) can receive multiple service provision messages from multiple devices in response to a voice command, “play music.” Based on the information contained in the multiple service provision messages, the group identification unit (125) can identify that a TV and a smartphone are included in the group corresponding to the received voice command. The service selection unit (126) can select all service provision messages received from the TV and smartphone included in the group and output the selected service provision messages.

[0342] According to another embodiment, the service selection unit (126) may select at least one service provision message based on service provision history information and / or device information received from the first group of devices. The service selection unit (126) may assign priority or weight to the service provision messages of the first group based on the service provision history information and / or device information, and select at least one service provision message according to the assigned priority or weight. For example, the service selection unit (126) may assign priority or weight to a plurality of service provision messages based on at least one of whether each device provides a service in response to the first voice command, the type of service provided by each device in response to the first voice command, the number of times the service provided by each device was selected for the first voice command, situation information when the service provided by each device was selected for the first voice command, and device information constituting each device.

[0343] As an example, the service selection unit (126) may select one service provision message with the highest priority or select multiple service provision messages with relatively high priority. As another example, the service selection unit (126) may select at least one service provision message with an assigned weight greater than or equal to a threshold.

[0344] For example, the group identification unit (125) may receive multiple service provision messages from multiple devices in response to a voice command, “play music.” Based on the information contained in the multiple service provision messages, the group identification unit (125) may identify that a TV and a smartphone are included in the group corresponding to the received voice command. The service selection unit (126) may assign priority to the devices based on the device information of the TV and the smartphone, according to the speaker performance of the devices. Based on the assigned priority, the service selection unit (126) may select the service provision message of the TV with the best speaker performance and output the selected service provision message.

[0345] A voice assistant service providing device (100) according to one embodiment may use a service recommendation model (143) for service recommendation to select at least one of the service providing messages of a first group of devices. For example, the service recommendation model (143) may be an artificial intelligence algorithm learned by a predetermined voice command, the service providing history of a plurality of devices regarding the predetermined voice command, and device information of each of the plurality of devices.

[0346] A service recommendation model (143) according to one embodiment can select at least one service provision message from a first group of service provision messages by comprehensively considering a first voice command, current situation information (e.g., location of a voice assistant service providing device (100), current time, etc.), service provision history information of each device, and device information.

[0347] A response message generation unit (127) of a voice assistant service providing device (100) according to one embodiment may generate a response message that includes information related to at least one service within at least one selected service providing message. The response message generation unit (127) may include a Natural Language Generation module that converts at least one selected service providing message into a human-understandable natural language, and a Text-to-Speech (TTS) module that converts the natural language in text form into a voice signal.

[0348] According to one embodiment, the response message generation unit (127) can generate and output one response message by listing the selected service provision messages when the service selection unit (126) selects two or more service provision messages.

[0349] For example, the service selection unit (126) can select service provision messages for TV and smartphone from among a plurality of service provision messages that respond to a voice command, “Play music.” In this case, the response message generation unit (127) can combine the service provision messages for TV and smartphone to output a response message, “You can play music through the TV or play music through the smartphone.” In this case, the user can listen to the output response message and decide which service to select.

[0350] In another embodiment, the response message generation unit (127) can generate and output one response message by combining the selected service provision messages when the service selection unit (126) selects two or more service provision messages.

[0351] For example, the service selection unit (126) can select service provision messages from a wearable device and an exercise machine from among a plurality of service provision messages that respond to a voice command, “Shall we try exercising?” In this case, the response message generation unit (127) can receive a service provision message from a wearable device containing result text, “Since you did strength training yesterday, I recommend aerobic exercise today,” and a service provision message from an exercise machine containing result text, “You can set the exercise to 30 minutes.” The response message generation unit (127) can combine the service provision messages from the wearable device and the exercise machine to generate and output a response message, “Since you did strength training yesterday, how about doing 30 minutes of aerobic exercise with the exercise machine?”

[0352] The response message generation unit (127) can learn result texts included in selected service provision messages and generate and output a single response message in which the result texts are combined. For example, the response message generation unit (127) can analyze the result texts and extract entities with the same or similar domains from the result texts. The response message generation unit (127) can generate a response message based on the extracted entities and the user's conversation history.

[0353] The response message generation unit (127) may use a response message generation model (145) for generating a response message to generate a single response message by combining selected service provision messages. For example, the response message generation model (145) may be an artificial intelligence algorithm learned from service provision messages received from multiple devices, service provision history information, and device information.

[0354] The response message generated by the response message generation unit (127) can be output through the speaker (150) of the voice assistant service providing device (100). As described above, the voice assistant service providing device (100) according to one embodiment allows the user to conveniently communicate with and control a large number of devices by suggesting the optimal service preferred by the user among various services provided by a plurality of devices.

[0355] Meanwhile, the user can listen to a response message responding to the first voice command and decide whether to accept the service proposed by the voice assistant service provider device (100). Based on the decision, the user can utter a second voice command or perform a predetermined action.

[0356] For example, in response to a first voice command, “Play music,” the voice assistant service provider (100) may output a response message through the speaker (150) saying, “Would you like to play music through the TV?” If the user decides to accept the service suggested by the voice assistant service provider (100), they may utter a second voice command, “Yes, play music through the TV.” If the user decides not to accept the service suggested by the voice assistant service provider (100), they may utter a second voice command, “No, play it through the smartphone.”

[0357] A voice assistant service providing device (100) according to one embodiment can determine whether the user has decided to accept a service proposed by the voice assistant service providing device (100) based on a second voice command received from the user.

[0358] A voice assistant service providing device (100) according to one embodiment can learn a service recommendation model (143) based on information related to a service selected by a user. The voice assistant service providing device (100) can continuously learn the service recommendation model (143) by the user's voice command in order to suggest an optimal service preferred by the user among various services provided by multiple devices.

[0359] FIG. 18 illustrates a specific block diagram of a voice assistant service providing device (100) according to one embodiment.

[0360] The voice assistant service provider (100) illustrated in FIG. 18 may include the same components as the voice assistant service provider (100) described in FIG. 16 and FIG. 17. For example, among the components illustrated in FIG. 18, the processor (2300) is the same as the processor (120) illustrated in FIG. 16 and FIG. 17, and the output unit (2200) is the same as the output unit (150) illustrated in FIG. 16 and FIG. 17.

[0361] The voice assistant service provider (100) illustrated in FIG. 18 can perform all the operations and functions of the voice assistant service provider (100) described in FIG. 1 to FIG. 17. Therefore, the components of the voice assistant service provider (100) that have not been described so far will be described below.

[0362] Referring to FIG. 18, a voice assistant service providing device (100) may include a user input unit (2100), an output unit (2200), a processor (2300), a sensing unit (2400), a communication unit (2500), an A / V input unit (2600), and a memory (2700).

[0363] The user input unit (2100) refers to a means for a user to input data for controlling a voice assistant service provider device (100). For example, the user input unit (2100) may include a key pad, a dome switch, a touch pad (contact capacitive method, pressure resistive method, infrared detection method, surface ultrasonic conduction method, integral tension measurement method, piezo effect method, etc.), a jog wheel, a jog switch, etc., but is not limited thereto. The user input unit (2100) can receive user input necessary to generate conversation information to be provided to the user.

[0364] The output unit (2200) can output an audio signal, a video signal, or a vibration signal, and the output unit (2200) may include a display unit (2210), an audio output unit (2220), and a vibration motor (2230).

[0365] The vibration motor (2230) can output a vibration signal. For example, the vibration motor (2230) can output a vibration signal corresponding to the output of audio data or video data (e.g., a call signal reception sound, a message reception sound, etc.).

[0366] The sensing unit (2400) can detect the state of the voice assistant service provider device (100) or the state of the surroundings of the voice assistant service provider device (100), and transmit the detected information to the processor (2300).

[0367] The sensing unit (2400) may include at least one of a magnetic sensor (2410), an acceleration sensor (2420), a temperature / humidity sensor (2430), an infrared sensor (2440), a gyroscope sensor (2450), a position sensor (e.g., GPS) (2460), a barometric pressure sensor (2470), a proximity sensor (2480), and an RGB sensor (illuminance sensor) (2490), but is not limited thereto. Since the function of each sensor can be intuitively inferred by a person skilled in the art from its name, a detailed description is omitted.

[0368] The communication unit (2500) may include components for performing communication with other devices. For example, the communication unit (2500) may include a short-range communication unit (2510), a mobile communication unit (2520), and a broadcast receiving unit (2530).

[0369] The short-range wireless communication unit (251) may include, but is not limited to, a Bluetooth communication unit, a BLE (Bluetooth Low Energy) communication unit, a near field communication unit, a WLAN (Wi-Fi) communication unit, a Zigbee communication unit, an infrared (IrDA, infrared Data Association) communication unit, a WFD (Wi-Fi Direct) communication unit, an ultra-wideband (UWB) communication unit, an Ant+ communication unit, etc.

[0370] The mobile communication unit (2520) transmits and receives wireless signals with at least one of a base station, an external terminal, and a server on a mobile communication network. Here, the wireless signal may include various forms of data such as voice call signals, video call call signals, or text / multimedia message transmission and reception.

[0371] The broadcast receiver (2530) receives broadcast signals and / or broadcast-related information from the outside through a broadcast channel. The broadcast channel may include a satellite channel and a terrestrial channel. Depending on the implementation example, the voice assistant service providing device (100) may not include a broadcast receiver (2530).

[0372] Additionally, the communication unit (2500) can transmit and receive information necessary to generate conversation information to be provided to the first user with the second interactive electronic device (3000), other devices, and a server.

[0373] The A / V (Audio / Video) input unit (2600) is for inputting audio or video signals and may include a camera (2610) and a microphone (2620), etc. The camera (2610) can obtain image frames, such as still images or video, through an image sensor in video call mode or shooting mode. Images captured through the image sensor can be processed through a processor (2300) or a separate image processing unit (not shown).

[0374] Image frames processed by the camera (2610) can be stored in memory (2700) or transmitted externally through the communication unit (2500). Two or more cameras (2610) may be provided depending on the configuration of the terminal.

[0375] The microphone (2620) receives an external acoustic signal and processes it into electrical voice data. For example, the microphone (2620) can receive an acoustic signal from an external device or a speaker. The microphone (2620) can use various noise removal algorithms to remove noise generated during the process of receiving the external acoustic signal.

[0376] The memory (2700) can store a program for processing and controlling the processor (2300), and can also store data that is input to or output from the voice assistant service provider (100).

[0377] The memory (2700) may include at least one type of storage medium among flash memory type, hard disk type, multimedia card micro type, card type memory (e.g., SD or XD memory, etc.), RAM, SRAM, ROM, EEPROM, PROM, magnetic memory, magnetic disk, and optical disk.

[0378] Programs stored in memory (2700) can be classified into multiple modules according to their functions, for example, UI modules, voice recognition modules, voice assistant service provision modules, touch screen modules, notification modules, etc. The UI module can provide specialized UI, GUI, etc. that are linked with the voice assistant service provision device (100) for each application. The voice recognition module can provide TTS functions, NLU functions, etc. for recognizing the user's voice commands. The voice assistant service provision module can provide a function to determine the device to which the user's voice command is to be transmitted in order to generate a response message for the user's voice command, analyze service provision messages received from the devices, and generate a response message based on the selected service provision message.

[0379] The touch screen module can detect a touch gesture on the user's touch screen and transmit information regarding the touch gesture to the processor (2300). In some embodiments, the touch screen module can recognize and analyze touch codes. The touch screen module may be configured as separate hardware including a controller.

[0380] The notification module can generate a signal to notify the occurrence of an event of the voice assistant service provider device (100). Examples of events occurring in the voice assistant service provider device (100) include receiving a call signal, receiving a message, inputting a key signal, and receiving a schedule notification. The notification module may output a notification signal in the form of a video signal through the display unit (2210), output a notification signal in the form of an audio signal through the sound output unit (2220), or output a notification signal in the form of a vibration signal through the vibration motor (2230).

[0381] Hereinafter, the configuration of a device (200) that operates in connection with a voice assistant service providing device (100) according to one embodiment is described. Each configuration of the device (200) described below can perform each step of the method of operation of the device (200) described above. Therefore, descriptions that overlap with the description above are omitted.

[0382] FIG. 19 is a block diagram of a device (200) according to one embodiment.

[0383] A device (200) according to one embodiment of the present disclosure may be a fixed terminal or a mobile terminal implemented as a computer device. A voice assistant service providing device (100) may be, for example, at least one of a smartphone, smart home appliance, wearable device, navigation, computer, laptop, digital broadcasting terminal, artificial intelligence speaker, PDA (Personal Digital Assistants), PMP (Portable Multimedia Player), and tablet PC, but is not limited thereto. The device (200) may communicate with the voice assistant service providing device (100), other devices and / or servers via a network using a wireless or wired communication method.

[0384] Referring to FIG. 19, the device (200) may include a receiver (210), a processor (220), a communication unit (230), a memory (240), and an output unit (250). Not all components shown in FIG. 19 are essential components of the device (200). The device (200) may be implemented by more components than those shown in FIG. 19, or by fewer components than those shown in FIG. 19. At least one component shown in FIG. 19 may perform the operation of FIG. 6 and FIG. 7. Therefore, descriptions that overlap with the descriptions above with reference to FIG. 6 and FIG. 7 are omitted.

[0385] A receiving unit (210) according to one embodiment of the present disclosure can receive voice commands from a user. For example, the receiving unit (210) can receive voice commands by converting external sound into electrical acoustic data using a microphone.

[0386] A memory (240) according to one embodiment of the present disclosure can store instructions for providing voice assistant services, various models used for voice recognition, neural networks, dictionary information, etc.

[0387] A processor (220) according to one embodiment of the present disclosure can provide voice assistant services according to various embodiments of the present disclosure by controlling a receiving unit (210), a communication unit (230), and an output unit (250) by executing one or more instructions stored in a memory (240). Although FIG. 19 is illustrated as having a device (200) that includes one processor (220), the present disclosure is not limited to the illustrated embodiment. The device (200) may include a plurality of processors. If the device (200) includes a plurality of processors, the operation and function of the processor (220) described below may be partially performed in the plurality of processes.

[0388] First, the processor (220) can receive information related to the first voice command from the voice assistant service provider (100) through the communication unit (230).

[0389] Information related to a first voice command received by the processor (220) of the device (200) from the voice assistant service providing device (100) may include an audio signal of the user's voice command itself or a result of processing the voice command. For example, the processor (220) may receive information regarding the result of interpreting the first text converted from the first voice command and the type of service requested by the first voice command as information related to the first voice command.

[0390] A processor (220) according to one embodiment can determine whether the device (200) can provide a service in response to information related to the first voice command.

[0391] The processor (220) can identify services that the device (200) can provide based on information about devices constituting the device (200) (e.g., screen, input interface, etc.), software information such as applications running on the device (200), and information stored in advance in the device (200) (e.g., current time information, and location information of the device, etc.).

[0392] For example, the processor (220) can identify services that the device (200) can provide based on at least one of information about the devices constituting the device (200), current time information, and location information of the device. The processor (220) can determine that the device (200) can provide a service in response to information related to the first voice command if, among the services identified as being available to the device (200), a service corresponding to the type of service requested by the first voice command is included.

[0393] If the processor (220) determines that it can provide a service in response to information related to the first voice command, the processor (220) may generate a service provision message to be transmitted to the voice assistant service provision device (100). The service provision message may include information related to a service proposed by the device (200) in response to information related to the first voice command received from the voice assistant service provision device (100).

[0394] For example, a service provision message may include at least one of whether the device can provide a service that responds to a voice command, whether the device is included in a predetermined group associated with the voice command, identification information of the device, identification information of a predetermined service provided by the device in response to a voice command, the type of the predetermined service, and identification information of an application used to provide the predetermined service.

[0395] A processor (220) of a device (200) according to one embodiment can transmit a service provision message and service provision history information to a voice assistant service providing device (100).

[0396] Service provision history information refers to information regarding past history related to services provided or proposed by the device (200) in response to information related to the first voice command. For example, the service provision history information may include at least one of whether the device can provide a service in response to the voice command, the number of times a predetermined service proposed by the device in response to the voice command was selected by the user, and situational information (e.g., time information, location information, etc.) when the predetermined service was selected in response to the voice command.

[0397] If the device (200) determines that it cannot provide a service in response to information related to the first voice command, the processor (220) may transmit a message through the communication unit (230) that service provision is impossible.

[0398] Additionally, a processor (220) according to one embodiment may receive information related to a service selected by a user from a voice assistant service providing device (100) through a communication unit (230). The processor (220) may update service provision history information stored in memory (240) based on the received information.

[0399] Meanwhile, the functions related to artificial intelligence according to the present disclosure are operated through a processor and memory. The processor may be composed of one or more processors. In this case, the one or more processors may be general-purpose processors such as CPUs, APs, and DSPs (Digital Signal Processors), graphics-dedicated processors such as GPUs and VPUs (Vision Processing Units), or artificial intelligence-dedicated processors such as NPUs. The one or more processors control the processing of input data according to predefined operation rules or artificial intelligence models stored in memory. Alternatively, if the one or more processors are artificial intelligence-dedicated processors, the artificial intelligence-dedicated processors may be designed with a hardware structure specialized for processing a specific artificial intelligence model.

[0400] The predefined rules of operation or artificial intelligence models are characterized by being created through learning. Here, being created through learning means that a predefined rules of operation or artificial intelligence models configured to perform desired characteristics (or objectives) are created by a basic artificial intelligence model being trained using multiple learning data by a learning algorithm. Such learning may be performed on the device itself where the artificial intelligence according to the present disclosure is executed, or it may be performed through a separate server and / or system. Examples of learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but are not limited to the examples described above.

[0401] An artificial intelligence model may be composed of multiple neural network layers. Each of the multiple neural network layers has multiple weight values ​​and performs neural network operations through operations between the results of previous layers and the multiple weights. The multiple weights possessed by the multiple neural network layers can be optimized based on the learning results of the artificial intelligence model. For example, the multiple weights may be updated so that the loss value or cost value obtained from the artificial intelligence model during the learning process is reduced or minimized. Artificial neural networks may include deep neural networks (DNNs), such as Convolutional Neural Networks (CNNs), Deep Neural Networks (DNNs), Recurrent Neural Networks (RNNs), Restricted Boltzmann Machines (RBMs), Deep Belief Networks (DBNs), Bidirectional Recurrent Deep Neural Networks (BRDNNs), or Deep Q-Networks, but are not limited to the examples mentioned above.

[0402] Additionally, the disclosed embodiments may be implemented as a software program including instructions stored on a computer-readable storage media.

[0403] A computer is a device capable of calling instructions stored from a storage medium and performing operations according to the disclosed embodiments according to the called instructions, and may include an image transmission device and an image reception device according to the disclosed embodiments.

[0404] Computer-readable storage media may be provided in the form of non-transitory storage media. Here, 'non-transitory' means merely that the storage medium does not contain a signal and is tangible, without distinguishing whether data is stored semi-permanently or temporarily on the storage medium.

[0405] Additionally, the electronic device or method according to the disclosed embodiments may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product.

[0406] A computer program product may include a software program and a computer-readable storage medium on which the software program is stored. For example, a computer program product may include a product in the form of a software program (e.g., a downloadable app) that is electronically distributed through a manufacturer of an electronic device or an electronic market (e.g., Google Play Store, App Store). For electronic distribution, at least a portion of the software program may be stored on a storage medium or temporarily created. In this case, the storage medium may be a server of the manufacturer, a server of the electronic market, or a storage medium of a relay server that temporarily stores the software program.

[0407] A computer program product may include a storage medium of a server or a storage medium of a terminal in a system composed of a server and a terminal (e.g., a voice assistant service provider or device). Alternatively, if a third device (e.g., a smartphone) is communicationally connected to the server or terminal, the computer program product may include a storage medium of the third device. Alternatively, the computer program product may include the S / W program itself, which is transmitted from the server to the terminal or the third device, or from the third device to the terminal.

[0408] In this case, one of the server, terminal, and third device may execute the computer program product to perform the method according to the disclosed embodiments. Alternatively, two or more of the server, terminal, and third device may execute the computer program product to perform the method according to the disclosed embodiments in a distributed manner.

[0409] For example, a server (e.g., a cloud server or an artificial intelligence server, etc.) can execute a computer program product stored on the server to control a terminal connected to the server in communication to perform a method according to the disclosed embodiments.

[0410] As another example, the third device may execute a computer program product to control a terminal connected to the third device in communication to perform a method according to the disclosed embodiment. As a specific example, the third device may remotely control a video transmission device or a video reception device to control the transmission or reception of a packing image.

[0411] When the third device executes a computer program product, the third device may download the computer program product from a server and execute the downloaded computer program product. Alternatively, the third device may execute a computer program product provided in a preloaded state to perform the method according to the disclosed embodiments.

Claims

Claim 1 A method for providing a voice assistant service performed by an electronic device, comprising: receiving a first voice command from a user; determining a plurality of devices to transmit the first voice command among a plurality of candidate devices; transmitting information related to the first voice command to the plurality of devices; receiving a plurality of service provision messages responding to the first voice command and service provision history information of the plurality of devices from each of the plurality of devices, wherein the service provision history information includes time information and location information in which a service provided in response to the first voice command was selected, and the plurality of devices are assigned priorities based on the time information and location information; selecting at least one of the plurality of service provision messages based on the priorities of the plurality of devices; and outputting a response message responding to the first voice command based on the selected result. Claim 2 The method according to claim 1, wherein the step of determining the plurality of devices comprises: a step of converting the first voice command into a first text; a step of interpreting the converted first text using a Natural Language Understanding (NLU) model; a step of determining the user's intent based on the interpretation result; and a step of determining the plurality of devices among the plurality of candidate devices to transmit the first voice command based on the relationship between the user's intent and the plurality of candidate devices. Claim 3 In claim 2, the step of determining the plurality of devices based on the relevance comprises: a step of obtaining device information regarding the plurality of candidate devices; a step of obtaining a plurality of probability values ​​representing the degree of relevance between the user's intention and the plurality of candidate devices based on the device information; and a step of determining the plurality of devices having probability values ​​greater than or equal to a threshold value among the plurality of candidate devices. Claim 4 A method according to claim 1, wherein a first service provision message received from a first device among the plurality of devices comprises at least one of whether the first device provides a service in response to the first voice command, whether the first device is included in a first group, identification information of the first device, identification information of the first service provided by the first device in response to the first voice command, the type of the first service, and identification information of an application used to provide the first service. Claim 5 A method according to claim 4, wherein the step of selecting at least one of the plurality of service provision messages comprises: a step of identifying devices included in the first group among the plurality of devices; and a step of selecting at least one of the service provision messages received from the identified devices based on service provision history information of the identified devices. Claim 6 A method according to claim 1, wherein the service provision history information comprises at least one of the number of times a first service proposed by the first device for the first voice command was selected by the user, and situation information when the first service was selected for the first voice command. Claim 7 A method according to claim 1, wherein the step of receiving the service provision messages and the service provision history information includes the step of receiving device information constituting each device from the plurality of devices together with the service provision messages and the service provision history information, and the step of selecting at least one of the plurality of service provision messages includes the step of selecting at least one of the plurality of service provision messages based on the device information of the plurality of devices and the service provision history information. Claim 8 A method according to claim 1, wherein the step of selecting at least one of the plurality of service provision messages includes the step of selecting at least one of the plurality of service provision messages using a service recommendation model, wherein the service recommendation model is an artificial intelligence algorithm learned by a predetermined voice command, the service provision history of the plurality of devices regarding the predetermined voice command, and device information of each of the plurality of devices. Claim 9 A method according to claim 1, wherein the step of selecting at least one of the plurality of service provision messages comprises: a step of assigning priority to the plurality of service provision messages based on at least one of whether each device provides a service in response to the first voice command, the type of service provided by each device in response to the first voice command, the number of times the service provided by each device was selected for the first voice command, situation information when the service provided by each device was selected for the first voice command, and device information constituting each device; and a step of selecting at least one of the plurality of service provision messages based on the assigned priority. Claim 10 A method according to claim 1, wherein the step of selecting at least one of the plurality of service provision messages comprises: a step of assigning priority to the plurality of service provision messages based on at least one of whether each device provides a service in response to the first voice command, the type of service provided by each device in response to the first voice command, the number of times the service provided by each device was selected for the first voice command, situation information when the service provided by each device was selected for the first voice command, and device information constituting each device; and a step of selecting two or more service provision messages among the plurality of service provision messages based on the assigned priority. Claim 11 A method according to claim 1, wherein a plurality of service provision messages received from a plurality of devices include information related to a service provided by each of the plurality of devices in response to a first voice command, and the step of outputting a response message in response to the first voice command comprises: a step of generating the response message to include information related to at least one service within the selected at least one service provision message; and a step of outputting the response message. Claim 12 In claim 1, the step of selecting at least one of the plurality of service provision messages comprises: identifying devices included in a first group among the plurality of devices based on the plurality of service provision messages; and selecting at least one of the service provision messages received from the identified devices based on service provision history information of the identified devices using a service recommendation model, wherein the method comprises: receiving a second voice command from the user; determining, based on the second voice command, a service selected by the user among the services provided by the plurality of devices in response to the first voice command; transmitting information related to the service selected by the user to the plurality of devices; and learning the service recommendation model using the information related to the service selected by the user. Claim 13 A voice assistant service providing device comprises: a receiving unit for receiving a voice command from a user; a communication unit; a memory for storing one or more commands; and at least one processor for executing a voice assistant program by executing the one or more commands. The at least one processor controls the receiving unit to receive a first voice command from the user, determines a plurality of devices among a plurality of candidate devices to transmit the first voice command, transmits information related to the first voice command to the plurality of devices, and controls the communication unit to receive a plurality of service providing messages responding to the first voice command and service providing history information of the plurality of devices from each of the plurality of devices, wherein the service providing history information includes time information and location information in which a service provided in response to the first voice command was selected, and the plurality of devices are assigned priorities based on the time information and location information. Based on the priorities of the plurality of devices, at least one of the plurality of service providing messages is selected, and based on the selected result, a response message responding to the first voice command is output. Claim 14 A voice assistant service providing device according to claim 13, wherein at least one processor converts the first voice command into a first text, interprets the converted first text, and transmits information regarding the interpretation result and the type of service requested by the first voice command as information related to the first voice command. Claim 15 A voice assistant service providing device according to claim 13, wherein a first service providing message received from a first device among the plurality of devices comprises at least one of whether the first device is included in a first group, identification information of the first device, identification information of a first service provided by the first device in response to the first voice command, the type of the first service, and identification information of an application used to provide the first service. Claim 16 A voice assistant service providing device according to claim 15, wherein the at least one processor identifies devices included in the first group among the plurality of devices and selects at least one of the service providing messages received from the identified devices based on service providing history information of the identified devices. Claim 17 A voice assistant service providing device according to claim 13, wherein the service provision history information comprises at least one of the number of times a first service proposed by the first device for the first voice command was selected by the user, and situation information when the first service was selected for the first voice command. Claim 18 A voice assistant service providing device according to claim 13, wherein the communication unit further receives device information constituting each device from the plurality of devices, and the at least one processor selects at least one of the plurality of service providing messages based on the device information of the plurality of devices and the service providing history information. Claim 19 A voice assistant service providing device according to claim 13, wherein the at least one processor selects at least one of the plurality of service providing messages using a service recommendation model, and the service recommendation model is an artificial intelligence algorithm learned by a predetermined voice command, the service providing history of the plurality of devices for the predetermined voice command, and device information of each of the plurality of devices. Claim 20 delete Claim 21 delete Claim 22 delete Claim 23 One or more computer-readable recording media storing a program that enables the method of claim 1 to be performed.

Images