Hub device, multi-device system including the hub device and multiple devices, and operating method of the hub device and the multi-device system

Abstract

A method for a hub device to execute and store a voice assistant model for controlling a device, the method comprising: receiving information about a first device connected to the hub device, requesting a voice assistant server to update a device determination model stored in the hub device; receiving the updated device determination model from the voice assistant server and storing the received updated device determination model; requesting information about a function determination model corresponding to the first device from the voice assistant server; receiving information about the function determination model corresponding to the first device from the voice assistant server and determining whether to store the function determination model in the memory based on the received information; and based on determining that the function determination model is stored in the hub device, storing the function determination model corresponding to the first device in the hub device.

Description

Technical Field

[0001] The present disclosure relates to a hub device, a multi-device system, and an operation method of the hub device and the multi-device system. The hub device includes: an artificial intelligence (AI) model for determining an operation execution device that executes an operation according to a user intention based on a voice input received from a user; and an AI model for controlling an operation executed by the determined operation execution device in a multi-device environment including the hub device and a plurality of devices. Background Art

[0002] With the development of multimedia technology and network technology, users can receive various services by using devices. In particular, with the development of speech recognition technology, users can input speech (e.g., utterances) to a device, and can receive a response message to the input speech through a service providing agent.

[0003] However, in a multi-device system including a plurality of devices such as a home network environment, when a user wants to receive a service by using a device other than a client device that interacts through voice input or the like, the user has to inconveniently select a device for providing the service. In particular, since the types of services that can be provided by a plurality of devices are different, a technology that can recognize an intention included in a user's voice input and effectively provide a corresponding service is needed.

[0004] In order to recognize an intention based on a user's voice input, artificial intelligence (AI) technology can be used, and a rule-based natural language understanding (NLU) technology can also be used. When a user's voice input is received by a hub device, since the hub device may not be able to directly select a device for providing a service based on the voice input and has to control the device by using a separate voice assistant service providing server, the user has to pay a network usage fee, and since the voice assistant service providing server is used, the response speed is reduced. To solve the above problems, the hub device can adopt an on-device model method, by which a voice assistant model for controlling a device for providing a service based on a voice input is stored in an internal memory. However, when using the on-device model method, due to the internal memory capacity, the remaining capacity of the random access memory (RAM), the processing speed, etc. of the hub device, the voice assistant model may not be stored. Summary of the Invention

[0005] [Technical Solution]

[0006] Provided are a multi-device system including a hub device and a plurality of devices, an operation method of the hub device and the multi-device system, and more particularly, a hub device, a multi-device system, and an operation method of the hub device and the multi-device system. The hub device receives a voice input from a user and stores at least some of the following voice assistant models in an internal memory of the hub device: a voice assistant model that automatically determines a device to perform an operation according to a user intention based on the received voice input; and a voice assistant model that provides information required to perform a service according to the determined device. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The above and other aspects and features of certain embodiments of the present disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings, in which,

[0008] like reference numerals denote identical structural elements, wherein:

[0009] Figure 1 is a block diagram of a multi-device system including a hub device, a voice assistant server, an Internet of Things (IoT) server, and a plurality of devices according to an embodiment;

[0010] Figure 2 is a block diagram of a hub device according to an embodiment;

[0011] Figure 3 is a block diagram of a voice assistant server according to an embodiment;

[0012] Figure 4 is a block diagram of an IoT server according to an embodiment;

[0013] Figure 5 is a block diagram of a plurality of devices according to an embodiment;

[0014] Figure 6 is a flowchart of a method of receiving and storing at least some voice assistant models from a voice assistant server, performed by a hub device according to an embodiment;

[0015] Figure 7 is a flowchart of a method of determining whether to store a function determination model based on information of a function determination model received from a voice assistant server, performed by a hub device according to an embodiment;

[0016] Figure 8 is a flowchart of a method of controlling an operation of a new device based on a voice input of a user, performed by a hub device according to an embodiment;

[0017] Figure 9 is a flowchart of an operation method of a hub device, a voice assistant server, an IoT server, and a new device according to an embodiment;

[0018] Figure 10 is a flowchart of an operation method of a hub device and a new device according to an embodiment;

[0019] Figure 11 is a flowchart of an operation method of a hub device, a voice assistant server, an IoT server, and a new device according to an embodiment;

[0020] Figure 12A is a conceptual diagram of a hub device and a listening device according to an embodiment;

[0021] Figure 12B is a conceptual diagram of a hub device and a second device according to an embodiment;

[0022] Figure 13 is a block diagram of a hub device according to an embodiment;

[0023] Figure 14 is a flowchart of an operation method of a hub device according to an embodiment;

[0024] Figure 15 is a flowchart of an operation method of a hub device according to an embodiment;

[0025] Figure 16 is a flowchart of an operation method of a hub device according to an embodiment;

[0026] Figure 17 is a schematic diagram of a multi-device system environment including a hub device, a voice assistant server, and multiple devices according to an embodiment;

[0027] Figure 18A is a schematic diagram of a voice assistant model executable by a hub device and a voice assistant server according to an embodiment; and

[0028] Figure 18B is a schematic diagram of a voice assistant model executable by a hub device and a voice assistant server according to an embodiment.

[0029] Best Mode

[0030] According to an aspect of the present disclosure, a method performed by a hub device for storing a voice assistant model for controlling a device, the method comprising: receiving information about a first device connected to the hub device, and after receiving the information about the first device, requesting a voice assistant server to update a device determination model stored in the hub device; receiving the updated device determination model from the voice assistant server and storing the received updated device determination model; requesting information about a function determination model corresponding to the first device from the voice assistant server; receiving information about the function determination model corresponding to the first device from the voice assistant server and determining whether to store the function determination model in a memory based on the received information; and based on determining that the function determination model is stored in the hub device, storing the function determination model corresponding to the first device in the hub device.

[0031] The method may further comprise: receiving access information for the function determination model corresponding to the first device based on determining that the function determination model is not stored in the hub device.

[0032] The access information may include at least one of the following: identification information of a voice assistant server storing the function determination model corresponding to the first device; location information; Internet protocol address information; media access control address; application programming interface information to which the function determination model in the voice assistant server is accessible; the language in which the function determination model is used; or identification information of the first device.

[0033] Determining whether to store the function determination model in the memory may include: determining whether to store the function determination model based on the resource state of the hub device and the information about the function determination model corresponding to the first device.

[0034] The hub device may be selected by the voice assistant server based on the resource state information about each of a plurality of devices pre-registered according to a user account.

[0035] The method may further comprise: selecting at least one candidate hub device from a plurality of devices pre-registered in an Internet of Things (IoT) server according to a user account logged in to the hub device; selecting one device from the at least one candidate hub device based on the usage history information and performance information of each of the at least one candidate hub device; and changing the hub device by replacing the hub device with one device selected from the at least one candidate hub device.

[0036] Selecting at least one candidate hub device may include: selecting at least one candidate hub device from a plurality of devices pre-registered in the IoT server based on at least one of power supply constancy, computing power, or power consumption of each of the plurality of devices pre-registered in the IoT server.

[0037] Selecting one device from at least one candidate hub device may include: obtaining usage frequency information about the hub device and each of the at least one candidate hub device by analyzing a usage history log database stored in the hub device; obtaining speech processing time information about the hub device and each of the at least one candidate hub device by analyzing a performance history log database stored in the hub device; and selecting a device for replacing the hub device based on the usage frequency information and the speech processing time information about the hub device and each of the at least one candidate hub device.

[0038] After receiving information about a first device connected to the hub device, an operation of selecting at least one candidate hub device from a plurality of devices pre-registered in the IoT server may be performed.

[0039] The method may further include: receiving a user input for selecting one device from a plurality of devices pre-registered in the IoT server according to a user account logged in to the hub device; and selecting one device from the plurality of devices pre-registered in the IoT server as the hub device based on the user input.

[0040] According to another aspect of the present disclosure, a hub device storing a voice assistant model for controlling a device, the hub device may include: a communication interface configured to perform data communication with at least one of a plurality of devices, a voice assistant server, or an Internet of Things (IoT) server; a memory configured to store a program including one or more instructions; and a processor configured to execute one or more instructions of the program stored in the memory to: receive information about a first device connected to the hub device, after receiving the information about the first device, request the voice assistant server to update a device determination model stored in the memory, and control the communication interface to receive the updated device determination model from the voice assistant server; store the received updated device determination model in the memory; request information about a function determination model corresponding to the first device from the voice assistant server, and control the communication interface to receive the information about the function determination model corresponding to the first device from the voice assistant server; determine whether to store the function determination model in the memory based on the received information; and based on determining that the function determination model is stored in the hub device, store the function determination model corresponding to the first device in the memory.

[0041] The processor may further be configured to execute one or more instructions to: based on determining that the function determination model is not stored in the hub device, control the communication interface to receive access information of the function determination model corresponding to the first device.

[0042] The access information may include at least one of the following: identification information of a voice assistant server storing a function determination model corresponding to the first device; location information; Internet protocol address information; media access control address; application programming interface information accessible to the function determination model in the voice assistant server; the language used by the function determination model; or identification information of the first device.

[0043] The processor may also be configured to execute one or more instructions to: determine whether to store the function determination model in the memory based on the resource state of the hub device and information about the function determination model corresponding to the first device.

[0044] The hub device may be selected by the voice assistant server based on the resource state information of each of the multiple devices pre-registered according to the user account.

[0045] The processor may also be configured to execute one or more instructions to: select at least one candidate hub device from the multiple devices pre-registered in the IoT server according to the user account logged into the hub device; select one device from the at least one candidate hub device based on the usage history information and performance information of each of the at least one candidate hub device; and change the hub device by replacing the hub device with one device selected from the at least one candidate hub device.

[0046] The processor may also be configured to execute one or more instructions to: select at least one candidate hub device from the multiple devices pre-registered in the IoT server based on at least one of the power supply constancy, computing power, or power consumption amount of each of the multiple devices pre-registered in the IoT server.

[0047] The device may further include: a usage history log database storing usage frequency information about each of the hub device and at least one candidate hub device; and a performance history log database storing speech processing time information about each of the hub device and at least one candidate hub device.

[0048] The processor may also be configured to execute one or more instructions to: obtain usage frequency information about each of the hub device and at least one candidate hub device by analyzing the usage history log database; obtain speech processing time information about each of the hub device and at least one candidate hub device by analyzing the performance history log database; and select a device for replacing the hub device based on the usage frequency information and speech processing time information about each of the hub device and at least one candidate hub device.

[0049] The device may further include a voice inputter configured to receive a user voice input for selecting one device from a plurality of devices pre-registered in an IoT server according to a user account logged in to the hub device. The processor may also be configured to execute one or more instructions to: convert the user's voice input received from the voice inputter into text by performing automatic speech recognition (ASR), and select, based on the user's voice input by interpreting the text using a natural language understanding (NLU) model, one device from the plurality of devices pre-registered in the IoT server as the hub device.

[0050] According to another aspect of the present disclosure, a method executed by a voice assistant server and a hub device, storing a voice assistant model for controlling a device, the method may include: after receiving information about a first device, the hub device requests the voice assistant server to update a device determination model stored in the hub device; based on receiving the request to update the device determination model, the voice assistant server updates the device determination model of the hub device; the hub device receives the updated device determination model from the voice assistant server and stores the received updated device determination model; the hub device requests information about a function determination model corresponding to the first device from the voice assistant server; the voice assistant server obtains information about the function determination model corresponding to the first device; the voice assistant server sends the information about the function determination model to the hub device; the hub device receives the information about the function determination model corresponding to the first device from the voice assistant server; and the hub device determines whether to store the function determination model in the memory based on the received information.

[0051] The method may further include: based on determining that the function determination model is stored in the hub device, the hub device stores the function determination model corresponding to the first device.

[0052] The method may further include: based on determining that the function determination model is not stored in the hub device, the hub device requests the voice assistant server to receive access information for the function determination model corresponding to the first device.

[0053] The access information may include at least one of the following: identification information of the voice assistant server storing the function determination model corresponding to the first device; location information; Internet protocol address information; media access control address; application programming interface information accessible to the function determination model in the voice assistant server; the language used for the function determination model; or identification information of the first device.

[0054] The voice assistant server may select the hub device based on resource status information about each of a plurality of devices pre-registered according to a user account.

[0055] According to another aspect of the present disclosure, a system may include a hub device and a voice assistant server. The hub device may include: a communication interface configured to perform data communication with the voice assistant server; a memory configured to store a program including one or more instructions; and a processor configured to execute one or more instructions of the program stored in the memory to: control the communication interface to receive information about a first device and request the voice assistant server to update a device determination model pre-stored in the memory; receive the updated device determination model from the voice assistant server and store the received updated device determination model in the memory; request information about a function determination model corresponding to the first device from the voice assistant server and control the communication interface to receive information about the function determination model corresponding to the first device from the voice assistant server; and determine whether to store the function determination model in the memory based on the received information. The voice assistant server may be configured to: update the device determination model pre-stored in the hub device based on a request to receive an updated device determination model from the hub device, obtain information about the function determination model corresponding to the first device, and send the obtained information to the hub device.

[0056] The processor may further be configured to execute one or more instructions to: store the function determination model corresponding to the first device in the memory based on determining that the function determination model is stored in the hub device.

[0057] The processor may further be configured to execute one or more instructions to: request access information about the function determination model corresponding to the first device from the voice assistant server and control the communication interface to receive the access information from the voice assistant server based on determining that the function determination model is not stored in the hub device.

[0058] The access information may include at least one of the following: identification information of the voice assistant server storing the function determination model corresponding to the first device; location information; Internet protocol address information; media access control address; application programming interface information accessible to the function determination model in the voice assistant server; the language used for the function determination model; or identification information of the first device.

[0059] The hub device may be selected by the voice assistant server based on resource status information about each of a plurality of devices pre-registered according to a user account. Detailed Description

[0060] This application is based on Korean Patent Application No. 10-2019-0138767, filed with the Korean Intellectual Property Office on November 1, 2019, and Korean Patent Application No. 10-2020-0065181, filed with the Korean Intellectual Property Office on May 29, 2020, and claims the benefit of their priority rights, and the above patent applications are incorporated herein by reference in their entirety.

[0061] Although the terms used herein are selected from currently widely used general terms in consideration of their functions in the present disclosure, these terms may vary according to the intention of those of ordinary skill in the art, precedent, or the emergence of new technologies. Additionally, in certain cases, these terms are optionally selected by the applicant of the present disclosure, and the meanings of these terms will be described in detail in the corresponding parts of the detailed description. Therefore, the terms used herein are not merely the names of the terms, but these terms are defined based on the meanings of the terms and the content throughout the present disclosure.

[0062] As used herein, unless the context clearly indicates otherwise, the singular forms "a", "an", and "the" are also intended to include the plural forms. The terms, including technical and scientific terms, used herein may have the same meanings as those commonly understood by those of ordinary skill in the art to which the present disclosure pertains.

[0063] Throughout the present disclosure, the expression "at least one of a, b, or c" indicates only a, only b, only c, both a and b, both a and c, both b and c, all of a, b, and c, or a variation thereof.

[0064] Throughout the present application, when a component "includes" an element, it should be understood that, as long as there is no specific contrary statement, the component also includes other elements rather than excluding other elements. Additionally, terms such as "unit", "module", etc., used herein indicate a unit that processes at least one function or operation, and the unit can be implemented as hardware or software, or a combination of hardware and software.

[0065] The expression "configured to (or set to)" used herein may be replaced, depending on the circumstances, by, for example, "adapted to", "capable of", "designed to", "suitable for", "manufactured to", or "able to". The expression "configured to (or set to)" does not necessarily mean "specially designed" in hardware. Instead, in some cases, the expression "a system configured to..." may mean that the system "is capable of..." together with other devices or components. For example, "a processor configured to (or set to) execute A, B, and C" may refer to a dedicated processor (e.g., an embedded processor) for performing the corresponding operations, or a general-purpose processor (e.g., a central processing unit (CPU) or an application processor (AP)) capable of performing the corresponding operations by executing one or more software programs stored in a memory.

[0066] As used herein, the "first natural language understanding (NLU) model" is a model trained to analyze text converted from voice input and determine an operation execution device based on the analysis result. The first NLU model can be used to determine an intention by interpreting the text and determine an operation execution device based on the intention.

[0067] As used herein, the "second NLU model" is a model trained to analyze text related to a specific device. The second NLU model can be a model trained to obtain operation information about an operation to be performed by a specific device by interpreting at least part of the text. The storage capacity of the second NLU model can be greater than the storage capacity of the first NLU model.

[0068] As used herein, the "voice assistant model" may include a device determination model and a function determination model. The "device determination model" is a model trained to determine an operation execution device from a plurality of devices pre-registered according to a user account based on the text analysis result of the first NLU model. The device determination model may include the first NLU model. The "function determination model" is a model for obtaining the following operation information: detailed operations for performing an operation according to the determined device function and the relationship between the detailed operations. The function determination model may include the second NLU model and an action planning management model.

[0069] As used herein, the "action planning management model" can be a model trained to manage operation information related to the detailed operations of a device to generate the detailed operations to be performed by the operation execution device and the order of executing the detailed operations. The action planning management model can manage operation information about the detailed operations of a device according to the device type and the relationship between the detailed operations.

[0070] As used herein, "intention" is information indicating a user intention determined by interpreting text. The intention as information indicating the intention of a user's utterance can be information indicating an operation execution device requested by the operating user. The intention can be determined by interpreting the text using an NLU model. For example, when the text converted from a user's voice input is "Play the movie Avengers on the TV", the intention can be "Content playback". Alternatively, when the text converted from a user's voice input is "Lower the air conditioner temperature to 18°C", the intention can be "Temperature control".

[0071] The intention can include not only information indicating the intention of a user's utterance (hereinafter referred to as intention information), but also a numerical value corresponding to the information indicating the user intention. The numerical value can indicate the probability that the text is related to the information indicating a specific intention. After interpreting the text using an NLU model, when multiple intention information indicating the user intention is obtained, the intention information with the largest numerical value among the multiple numerical values corresponding to the multiple intention information can be determined as the intention.

[0072] The term "operation" of the device used in this document may refer to at least one action performed by the device when the device executes a specific function. An operation may indicate at least one action performed by the device when the device executes an application. For example, an operation may indicate video playback, music playback, email creation, weather information reception, news information display, gaming, and shooting, for example, when the device executes an application. However, the operations are not limited to the above examples.

[0073] The operation of the device may be performed based on information about the detailed operations output from the action planning management model. The device may perform at least one action by executing functions corresponding to the detailed operations output from the action planning management model. The device may store instructions for executing the functions corresponding to the detailed operations, and when the detailed operation is determined, the device may determine the instructions corresponding to the detailed operation, and may execute a specific function by executing the instructions.

[0074] In addition, the device may store instructions for executing the application corresponding to the detailed operation. The instructions for executing the application may include instructions for executing the application itself and instructions for executing the detailed functions that make up the application. When the detailed operation is determined, the device may execute the application by executing the instructions for executing the application corresponding to the detailed operation, and may execute the detailed function by executing the instructions for executing the detailed functions of the application corresponding to the detailed operation.

[0075] The "operation information" used in this document may be information related to the detailed operations to be determined by the device, the relationship between each detailed operation and another detailed operation, and the order of executing the detailed operations. When the first operation is to be executed, the relationship between each detailed operation and another detailed operation includes information about the second operation that must be executed before executing the first operation. For example, when the operation to be executed is "music playback", "power on" may be another detailed operation that must be executed before executing "music playback". The operation information may include, but is not limited to, the functions executed by the operation execution device to execute a specific operation, the execution order of the functions, the input values required to execute the functions, and the output values output as a result of the function execution.

[0076] The term "operation execution device" used in this document refers to a device that determines a device for executing an operation from multiple devices based on the intention obtained from the text. The text may be analyzed by using a first NLU model, and the operation execution device may be determined based on the analysis result. The operation execution device may perform at least one action by executing functions corresponding to the detailed operations output from the action planning management model. The operation execution device may perform an operation based on the operation information.

[0077] As used herein, an "Internet of Things (IoT) server" is a server that obtains, stores, and manages device information about each of a plurality of devices. The IoT server can obtain, determine, or generate control commands for controlling the devices by using the stored device information. The IoT server can send the control commands to the devices determined to perform operations based on operation information. The IoT server can be implemented as a hardware device independent of the "server" of the present disclosure, but is not limited thereto. The IoT server can be an element of the "voice assistant server" of the present disclosure, or can be a server designed to be classified as software.

[0078] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art can easily implement and practice the present disclosure. However, the present disclosure can be implemented in many different forms and should not be construed as limited to the embodiments of the present disclosure set forth herein.

[0079] Reference will now be made in detail to embodiments of the present disclosure, examples of which are illustrated in the accompanying drawings.

[0080] Figure 1 is a multi-device system according to an embodiment, including a hub device 1000, a voice assistant server 2000, an IoT server 3000, and a plurality of devices 4000.

[0081] In Figure 1 , for ease of explanation, only the basic elements for describing the operations of the hub device 1000, the voice assistant server 2000, the IoT server 3000, and the plurality of devices 4000 are shown. The elements included in the hub device 1000, the voice assistant server 2000, the IoT server 3000, and the plurality of devices 4000 are not limited to Figure 1 those shown in

[0082] Figure 1 The reference numerals S1 to S16 marked by arrows in

[0083] indicate that data moves, is sent, and is received between multiple entities through a network. The numbers attached to the English letter S in S1 to S16 are used for identification and are independent of the order in which data moves, is sent, and is received. Figure 1, the hub device 1000, the voice assistant server 2000, the IoT server 3000, and the plurality of devices 4000 can be connected to each other and perform communication by using a wired or wireless communication method. In an embodiment, the hub device 1000 and the plurality of devices 4000 can be directly connected to each other through a communication network, but the present disclosure is not limited thereto. The hub device 1000 and the plurality of devices 4000 can be connected to the voice assistant server 2000, and the hub device 1000 can be connected to the plurality of devices 4000 through the voice assistant server 2000. In addition, the hub device 1000 and the plurality of devices 4000 can be connected to the IoT server 3000. In another embodiment, each of the hub device 1000 and the plurality of devices 4000 can be connected to the voice assistant server 2000 through a communication network and can be connected to the IoT server 3000 through the voice assistant server 2000.

[0084] The hub device 1000, the voice assistant server 2000, the IoT server 3000, and the plurality of devices 4000 can be connected through a local area network (LAN), a wide area network (WAN), a value-added network (VAN), a mobile radio communication network, a satellite communication network, or a combination thereof. Examples of the wireless communication method can include but are not limited to Wi-Fi, Bluetooth, Bluetooth Low Energy (BLE), Zigbee, Wi-Fi Direct (WFD), Ultra-Wideband (UWB), Infrared Data Association (IrDA), and Near Field Communication (NFC).

[0085] The hub device 1000 is a device that receives a voice input from a user and controls at least one of the plurality of devices 4000 based on the received voice input. The hub device 1000 can be a listening device that receives a voice input from a user. For example, the hub device 1000 can be but is not limited to a voice assistant speaker.

[0086] The hub device 1000 can be selected by the voice assistant server 2000 based on the resource status information of each of the plurality of devices pre-registered in the IoT server 3000 according to the user account.

[0087] At least one of the plurality of devices 4000 may be an operation execution device that executes a specific operation by receiving a control command from the collector device 1000 or the IoT server 3000. The plurality of devices 4000 may be devices that log in using the same user account as the hub device 1000 and are pre-registered in the IoT server 3000 using the user account of the hub device 1000. However, the present disclosure is not limited thereto, and the plurality of devices 4000 may include a new device 4300 newly registered in the IoT server 3000 according to a user account. The new device 4300 may be a device newly acquired by the user of the hub device 1000 and registered in the IoT server 3000 by logging in using the same user account as the hub device 1000 and other devices (e.g., the first device 4100 and the second device 4200).

[0088] At least one of the plurality of devices 4000 may receive a control command (S12, S14, and S16) from the IoT server 3000, or may receive at least a part of the text converted from a voice input from the hub device 1000 (S3 and S5). At least one of the plurality of devices 4000 may receive a control command (S12, S14, and S16) from the IoT server 3000 without receiving at least a part of the text from the hub device 1000.

[0089] The hub device 1000 may include a device determination model 1340, where the device determination model 1340 determines a device for executing an operation based on a user's voice input. The device determination model 1340 may determine an operation execution device from among the plurality of devices 4000 registered according to a user account. In an embodiment, the hub device 1000 may receive device information including at least one of the following from the voice assistant server 2000: identification information (e.g., device id information) of each of the plurality of devices 4000, the device type of each of the plurality of devices 4000, the function execution ability of each of the plurality of devices 4000, location information, or status information (S2). The hub device 1000 may determine, from among the plurality of devices 4000, a device for executing an operation according to the user's voice input by using data on the device determination model 1340 based on the received device information.

[0090] In another embodiment, the hub device 1000 may directly receive device information including at least one of the following from a plurality of devices: identification information (e.g., device ID information) of each of the plurality of devices 4000, the device type of each of the plurality of devices 4000, the function execution ability of each of the plurality of devices 4000, location information, or status information. The hub device 1000 may identify the plurality of devices connected to the access point connected to the hub device 1000, and may obtain the device information by requesting the device information from the identified plurality of devices. In this case, the plurality of devices may include an SW module configured to send the device information according to the request of the hub device 1000. Additionally, the hub device 1000 may obtain the device information by receiving the device information sent by the plurality of devices. In this case, the plurality of devices may include an SW module configured to identify the hub device 1000 connected to the access point connected to the plurality of devices, and an SW module configured to send the device information to the identified hub device 1000.

[0091] When the hub device 1000 obtains information related to a new device (first device) connected to the hub device 1000, the hub device 1000 may request the voice assistant server 2000 to update the device determination model stored in the hub device 1000. In an embodiment, the hub device 1000 may analyze the text related to the new device based on the voice input of the user based on the device information of the new device received from the IoT server 3000, and may send a query signal for requesting to update the device determination model stored in the memory 1300 (see Figure 2 ) to the voice assistant server 2000 to determine the new device as an operation execution device. In this case, the hub device 1000 may send at least the user account information and the identification information (e.g., the ID information of the hub device 1000) of the hub device 1000 together with the query signal to the voice assistant server 2000.

[0092] In response to the request of the hub device 1000, the hub device 1000 receives the updated device determination model from the voice assistant server 2000, and stores the received updated device determination model. In an embodiment, the hub device 1000 may analyze the user voice input regarding the new device, and as a result of the analysis, may download the updated device determination model from the voice assistant server 2000 to determine the new device as an operation execution device, and may store the updated device determination model in the memory 1300 (see Figure 2 ) inside the hub device 1000.

[0093] Additionally, the function determination model corresponding to the operation execution device determined by the hub device 1000 may be stored in the memory 1300 of the hub device 1000 (seeFigure 2 ) can be stored in the operation execution device itself or can be stored in the memory 2300 of the voice assistant server 2000 (see Figure 3 ). The memory 1300 of the hub device 1000 is a non-volatile memory. A non-volatile memory refers to a storage medium that can store and retain information even when not powered and can use the stored information when powered, and may include, for example, at least one of flash memory, a hard disk, a solid state drive (SSD), a multimedia card micro type, a card type memory (such as an SD or XD memory), a read-only memory (ROM), a magnetic memory, a magnetic disk, or an optical disk. The term "function determination model" corresponding to each device refers to a model of operation information regarding detailed operations performed according to the determined device functions and the relationships between the detailed operations.

[0094] The hub device 1000 can monitor the resource status of the hub device 1000 and the multiple devices 4000, and can obtain resource status information by using the resource tracking module 1350. The resource status information includes information related to the usage status of the processors 1200, 4120, 4220, and 4320 and the memories 1300, 4130, 4230, and 4330 of the hub device 1000 and the multiple devices 4000. The resource status information may include, for example, at least one of the remaining capacity, the average RAM usage capacity, or the average processor occupancy information of the memories 1300, 4130, 4230, and 4330 of the hub device 1000 and the multiple devices 4000. The resource tracking module 1350 can provide the obtained resource status information to the function determination model management module 1360.

[0095] The hub device 1000 can determine whether to store and process the function determination model 1370 in the memories 1300, 4130, 4230, and 4330 of the hub device 1000 and the multiple devices 4000 by using the function determination model management module 1360. When there are multiple function determination models 1370, the hub device 1000 can determine whether to store and process each of the multiple function determination models 1370 in the memories 1300, 4130, 4230, and 4330 of the hub device 1000 and the multiple devices 4000. The function determination model management module 1360 can compare the information (such as capacity information) of the function determination model received from the voice assistant server 2000 with the resource status information received from the resource tracking module 1350, and can analyze the compared information. The function determination model management module 1360 can determine whether to download the function determination model from the voice assistant server 2000 and store the function determination model in the memories 1300, 4130, 4230, and 4330 of the hub device 1000 and the multiple devices 4000 by using the comparison / analysis result.

[0096] In an embodiment, based on the function determination model management module 1360, it is determined to download the function determination model from the voice assistant server 2000 and store the function determination model in the memory 1300. The hub device 1000 can send a query to the voice assistant server 2000 for requesting to send the function determination model, can receive the function determination model from the voice assistant server 2000, and can store the function determination model in the memory 1300.

[0097] In another embodiment, based on the function determination model management module 1360, it is determined to download the function determination model from the voice assistant server 2000 and store the function determination model in the memory 4130, 4230, or 4330 of at least one of the multiple devices 4000 other than the hub device 1000. The hub device 1000 can send a query to the voice assistant server 2000 for requesting to send the function determination model, can receive the function determination model from the voice assistant server 2000, and can send the function determination model to at least one of the multiple devices 4000. The voice assistant server 2000 can send the function determination model to at least one of the multiple devices 4000 other than the hub device 1000.

[0098] In another embodiment, based on the function determination model management module 1360, it is determined not to download the function determination model from the voice assistant server 2000. The hub device 1000 can receive the access information of the function determination model pre-stored in the voice assistant server 2000, and can store the received access information in the function determination model information database 1362.

[0099] The function determination model information database 1362 can store the capacity information and access information of the function determination model corresponding to the device newly registered or registered by using the same user account as the hub device 1000 among the multiple function determination models 2362, 2364, 2366, and 2368 pre-stored in the voice assistant server 2000. The access information can include at least one of the following: the identification information (e.g., server id) of the voice assistant server 2000 storing each function determination model, location information, Internet Protocol (IP) address information, Media Access Control (MAC) address, the application programming interface (API) information accessible to each function determination model in the voice assistant server 2000, the language used for each function determination model, or the identification information of the corresponding device.

[0100] In an embodiment, when registering a new device using a user account, the hub device 1000 receives information (e.g., capacity information) about a function determination model corresponding to the new device from the voice assistant server 2000 by using the function determination model management module 1360. The hub device 1000 may determine whether to download the function determination model from the voice assistant server 2000 and store the function determination model in the memory 1300 based on the received information about the function determination model (see Figure 2 ). Based on determining to store the function determination model in the memory 1300, the hub device 1000 may download the function determination model corresponding to the new device from the voice assistant server 2000 and store the function determination model in the memory 1300. Based on determining not to store the function determination model, the hub device 1000 may store access information of the function determination model corresponding to the new device in the function determination model information database 1362.

[0101] The hub device 1000 may determine the function of the hub device 1000 itself and store a first function determination model 1372 for performing operations according to the determined function and a second function determination model 1374 corresponding to at least one of the plurality of devices 4000. For example, when the hub device 1000 is a voice assistant speaker, the hub device 1000 may store the first function determination model 1372 (e.g., a function determination model of the speaker), where the first function determination model 1372 is used to obtain operation information about detailed operations for performing the function of the hub device 1000 and the relationships between the detailed operations. Alternatively, the hub device 1000 may store the second function determination model 1374 (e.g., a function determination model of the TV), where the second function determination model 1374 is used to obtain operation information about detailed operations corresponding to the TV and the relationships between the detailed operations. The TV may be a device pre-registered in the IoT server 3000 using the same user account as the hub device 1000.

[0102] The first function determination model 1372 (e.g., a function determination model of the speaker) and the second function determination model 1374 (e.g., a function determination model of the TV) may respectively include a second NLU model 1372a and 1374a and an action planning management model 1372b and 1374b. The second NLU models 1372a and 1374a and the action planning management models 1372b and 1374b will be described in detail with reference to Figure 2 the detailed description.

[0103] The voice assistant server 2000 can determine an operation execution device for performing an operation for a user intention based on the text received from the hub device 1000. The voice assistant server 2000 can receive user account information from the hub device 1000 (S1). When the voice assistant server 2000 receives the user account information from the hub device 1000, the voice assistant server 2000 can send a query for requesting device information about a plurality of devices 4000 pre-registered according to the received user account information to the IoT server 3000 (S9), and can receive the device information about the plurality of devices 4000 from the IoT server 3000 (S10). The device information may include at least one of the following: identification information (e.g., device id information) of each of the plurality of devices 4000, device type of each of the plurality of devices 4000, function execution ability of each of the plurality of devices 4000, location information, or status information. The voice assistant server 2000 can send the device information received from the IoT server 3000 to the hub device 1000 (S2).

[0104] The voice assistant server 2000 may include a device determination model 2330, a voice assistant model update module 2340, a device-side model update module 2350, and a plurality of function determination models 2362, 2364, 2366, and 2368. The voice assistant server 2000 can select a function determination model corresponding to at least part of the text received from the hub device 1000 from the plurality of function determination models 2362, 2364, 2366, and 2368 by using the device determination model 2330, and can obtain operation information for the operation execution device to perform an operation by using the selected function determination model. The voice assistant server 2000 can send the operation information to the IoT server 3000 (S9).

[0105] The voice assistant server 2000 can update the voice assistant model by using the voice assistant model update module 2340. When a new device is newly registered in the IoT server 3000 or a new function is added to an existing device by using a user account, the voice assistant server 2000 can update the device determination model 2330 and the function determination model 2368 by using the voice assistant model update module 2340. For example, when the function determination model 2364 corresponding to the second device 4200 is updated, for example, when the function of the second device 4200 is added or changed, the voice assistant model update module 2340 can update the device determination model 2330 to a new model by learning or the like, so that the device determination model 2330 determines the second device 4200 as an operation execution device related to the updated function by interpreting the updated function. In an embodiment, the voice assistant model update module 2340 can update the first NLU model 2332 to a new model by learning or the like, so that the first NLU model 2332 of the device determination model 2330 interprets the text related to the newly updated function.

[0106] For example, when a new device 4300 is registered in the IoT server 3000 by using a user account, the voice assistant model update module 2340 can receive the identification information of the new device 4300 and the function determination model 2368 corresponding to the new device 4300 from the IoT server 3000, and can also store the received function determination model 2368 in the function determination models 2362, 2364, and 2366 in the memory 2300 (see Figure 3 ). The voice assistant model update module 2340 can update the device determination model 2330 to a new model by learning or the like, so that the device determination model 2330 interprets the intention from the text, and determines the new device 4300 as an operation execution device related to the intention as a result of the interpretation. In an embodiment, the voice assistant model update module 2340 can update the first NLU model 2332 to a new model by learning or the like, so that the first NLU model 2332 of the device determination model 2330 interprets the text related to the new device 4300.

[0107] The voice assistant server 2000 can train a voice assistant model according to a user account by using an on-device model update module 2350, and can determine whether to send at least some of the voice assistant models updated by learning to the hub device 1000. In another embodiment, the voice assistant server 2000 can determine to send at least some of the voice assistant models updated by learning to at least one of a plurality of devices 4000 other than the hub device 1000. The on-device model update module 2350 can include an on-device model information database 2352 that stores configuration information of the voice assistant model according to the user account. The on-device model information database 2352 can store at least one of the following: for example, user account information, identification information about a plurality of devices registered according to the user account, identification information of a function determination model corresponding to each of the plurality of devices, version information of the function determination model, version information of the device determination model, or information about a device type that can be determined as an operation execution device by the device determination model.

[0108] When the voice assistant server 2000 determines to send the updated device determination model 2330 to the hub device 1000 by using the on-device model update module 2350, the voice assistant server 2000 can send the updated device determination model 2330 to the hub device 1000 through a communication interface 2100 (see Figure 3 ). In an embodiment, when the voice assistant server 2000 receives a signal (S1) from the hub device 1000 requesting to send a function determination model 2368 corresponding to a new device, the voice assistant server 2000 can send the function determination model 2368 corresponding to the new device to the hub device 1000 by using the on-device model update module 2350 (S2). In another embodiment, the voice assistant server 2000 can send the function determination model 2368 corresponding to the new device to at least one of the plurality of devices 4000 by using the on-device model update module 2350.

[0109] Reference will be made to Figure 3 the detailed description of each of the plurality of function determination models 2362, 2364, 2366, and 2368 stored in the voice assistant server 2000.

[0110] The IoT server 3000 can be connected via a network and can store information about multiple devices 4000 that are pre-registered with user accounts using the hub device 1000. In an embodiment, the IoT server 3000 can receive at least one of the following: user account information for logging into each of the multiple devices 4000, identification information (e.g., device ID information) for each of the multiple devices 4000, the device type of each of the multiple devices 4000, or information about the function execution capabilities of each of the multiple devices 4000 (S11, S13, and S15). In an embodiment, the IoT server 3000 can receive status information about the on / off state or the operations being performed for each of the multiple devices 4000 from the multiple devices 4000 (S11, S13, and S15). The IoT server 3000 can store the device information and status information received from the multiple devices 4000.

[0111] The IoT server 3000 can send the device information and status information received from each of the multiple devices 4000 to the voice assistant server 2000 (S10).

[0112] The IoT server 3000 can generate control commands that can be read and executed by the operation execution device based on the operation information received from the voice assistant server 2000. The IoT server 3000 can send the control commands to the device among the multiple devices 4000 that is determined to be the operation execution device (S12, S14, and S16).

[0113] Reference will be made to Figure 4 describe the components of the IoT server 3000 in detail.

[0114] In Figure 1 , the multiple devices 4000 can include a first device 4100, a second device 4200, and a new device 4300. Although in Figure 1 the first device 4100 is an air conditioner, the second device 4200 is a TV, and the new device 4300 is an air purifier, the present disclosure is not limited thereto. The multiple devices 4000 can include not only air conditioners, TVs, and air purifiers, but also household appliances such as robot cleaners, washing machines, ovens, microwave ovens, scales, refrigerators, or electronic photo frames, and mobile devices such as smartphones, tablet personal computers (PCs), mobile phones, video phones, e-book readers, desktop PCs, laptop PCs, netbook computers, workstations, servers, personal digital assistants (PDAs), portable multimedia players (PMPs), MP3 players, mobile medical devices, cameras, or wearable devices.

[0115] At least one of the plurality of devices 4000 may store a function determination model by itself. For example, the memory 4130 of the first device 4100 may store a function determination model 4132, where the function determination model 4132 is used to obtain operation information about the detailed operations and the relationships between the detailed operations required for the first device 4100 to perform an operation determined according to a user's voice input, and to generate a control command based on the operation information.

[0116] Neither the second device 4200 nor the new device 4300 among the plurality of devices 4000 stores a function determination model. However, the present disclosure is not limited to this. In an embodiment, the function determination model corresponding to the new device 4300 may be stored in the internal memory of the new device 4300 itself.

[0117] At least one of the plurality of devices 4000 may send information on whether the device itself stores a function determination device corresponding to each of the plurality of devices 4000 to the hub device 1000 (S4, S6, and S8).

[0118] Figure 2 It is a block diagram of the hub device 1000 according to an embodiment.

[0119] The hub device 1000 is a device that receives a user's voice input and controls at least one of the plurality of devices 4000 based on the received voice input. The hub device 1000 may be a listening device that receives a voice input from a user.

[0120] Reference Figure 2 , the hub device 1000 may at least include a microphone 1100, a processor 1200, a memory 1300, and a communication interface 1400. The hub device 1000 may receive a voice input (e.g., a user's words) from a user through the microphone 1100, and may obtain a voice signal from the received voice input. In an embodiment, the processor 1200 of the hub device 1000 may convert the sound received through the microphone 1100 into an acoustic signal, and may obtain a voice signal by removing noise (e.g., non-speech components) from the acoustic signal.

[0121] Although not shown in Figure 2 , the hub device 1000 may include a voice recognition module that has a function of detecting a specified voice input (e.g., a wake-up input such as "Hi, Bixby" or "OK, Google") or a function of preprocessing a voice signal obtained from a partial voice input.

[0122] The processor 1200 can execute one or more instructions of a program stored in the memory 1300. The processor 1200 may include hardware components that perform arithmetic, logical, and input / output operations, as well as signal processing. The processor 1200 may include at least one of the following components, including but not limited to: a central processing unit, a microprocessor, a graphics processing unit, an application specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), or a field programmable gate array (FPGA).

[0123] A program including instructions for controlling a plurality of devices 4000 based on a voice input of a user received through the microphone 1100 may be stored in the memory 1300. Instructions and program codes readable by the processor 1200 may be stored in the memory 1300. In the following embodiments, the processor 1200 may be implemented by executing the instructions or codes stored in the memory.

[0124] Data regarding an automatic speech recognition (ASR) module 1310, data regarding a natural language generation (NLG) module 1320, data regarding a text-to-speech (TTS) module 1330, data regarding a device determination model 1340, data regarding a resource tracking module 1350, data regarding a function determination model management module 1360, and data corresponding to each of a plurality of function determination models 1370 may be stored in the memory 1300.

[0125] The memory 1300 may include non-volatile memory. The memory 1300 may include at least one of the following various types of storage media: for example, flash type, hard disk type, solid state drive (SSD), multimedia card micro type, card type memory (e.g., SD or XD memory), random access memory (RAM), static random access memory (SRAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), programmable read only memory (PROM), magnetic memory, magnetic disks, and optical disks.

[0126] The processor 1200 can perform automatic speech recognition (ASR) by using the data regarding the ASR module 1310 stored in the memory 1300, and convert the voice signal received through the microphone 1100 into text.

[0127] The processor 1200 can analyze text by using data about the device determination model 1340 stored in the memory 1300, and can determine an operation execution device from multiple devices 4000 based on the analysis result of the text. The device determination model 1340 can include a first NLU model 1342. In an embodiment, the processor 1200 can analyze text by using data about the first NLU model 1342 included in the device determination model 1340, and can determine an operation execution device for performing an operation according to the user intention from multiple devices 4000 based on the analysis result of the text.

[0128] The first NLU model 1342 is a model trained to analyze text converted from voice input and determine an operation execution device based on the analysis result. The first NLU model 1342 can be used to determine an intention by interpreting the text and determine an operation execution device based on the intention.

[0129] In an embodiment, the processor 1200 can parse the text in units of morphemes, words, or phrases by using data about the first NLU model 1342 stored in the memory 1300, and can infer the meaning of a word extracted from the parsed text by using language features (e.g., grammatical components) of the morpheme, word, or phrase. The processor 1200 can compare the inferred meaning of the word with a predefined intention provided by the first NLU model 1342, and can determine an intention corresponding to the inferred meaning of the word.

[0130] The processor 1200 can determine a device related to the intention identified from the text as an operation execution device based on a matching model for determining the relationship between the intention and the device. In an embodiment, the matching model can be included in the data about the device determination model 1340 stored in the memory 1300, and can be obtained by learning through a rule-based system, but the present disclosure is not limited thereto.

[0131] In an embodiment, the processor 1200 can apply the matching model to the intention to obtain multiple numerical values indicating the degree of relationship between the intention and multiple devices 4000, and can determine the device having the maximum value among the obtained multiple numerical values as the final operation execution device. For example, when the intention is related to each of a first device 4100 (see Figure 1 ) and a second device 4200 (see Figure 1 ), the processor 1200 can obtain a first numerical value indicating the degree of relationship between the intention and the first device 4100 and a second numerical value indicating the degree of relationship between the intention and the second device 4200, and can determine the first device 4100 having a larger value among the first numerical value and the second numerical value as the operation execution device.

[0132] For example, when the hub device 1000 receives a voice input from a user saying "Lower the set temperature by 2°C due to heat", the processor 1200 can perform ASR to convert the voice input into text by using data regarding the ASR module 1310, and can analyze the converted text by using data related to the first NLU model 1342 to obtain an intention corresponding to "set temperature adjustment". The processor 1200 can obtain a first numerical value indicating the degree of relationship between the intention of "set temperature adjustment" and the first device 4100 which is an air conditioner, a second numerical value indicating the degree of relationship between the intention of "set temperature adjustment" and the second device 4200 which is a TV, and a third numerical value indicating the degree of relationship between the intention of "set temperature adjustment" and the new device 4300 which is an air purifier (see Figure 1 ). The processor 1200 can determine the first device 4100 as the operation execution device related to "set temperature adjustment" by using the first numerical value which is the maximum value among the obtained numerical values.

[0133] As another example, when the hub device 1000 receives a voice input from a user saying "Play the movie Avengers", the processor 1200 can analyze the text converted from the voice input, and can obtain an intention corresponding to "content playback". The processor 1200 can determine the second device 4200 as the operation execution device related to "content playback" based on the second numerical information which is the maximum value among a first numerical value indicating the degree of relationship between the intention of "content playback" and the first device 4100 which is an air conditioner, a second numerical value indicating the degree of relationship between the intention of "content playback" and the second device 4200 which is a TV, and a third numerical value indicating the degree of relationship between the intention of "content playback" and the new device 4300 which is an air purifier, and which is calculated by using the matching model.

[0134] However, the present disclosure is not limited to the above examples, and the processor 1200 can arrange the numerical values indicating the degree of relationship between the intention and multiple devices in ascending order, and can determine a predetermined number of devices as operation execution devices. In an embodiment, the processor 1200 can determine a device whose numerical value indicating the degree of relationship is equal to or greater than a certain threshold as the operation execution device related to the intention. In this case, multiple devices can be determined as operation execution devices.

[0135] Although the processor 1200 can train a matching model between an intention and an operation execution device by using, for example, a rule-based system, the present disclosure is not limited thereto. The artificial intelligence (AI) model used by the processor 1200 can be, for example, a neural network-based system (e.g., a convolutional neural network (CNN) or a recurrent neural network (RNN)), a support vector machine (SVM), linear regression, logistic regression, naive Bayes, random forest, decision tree, or k-nearest neighbor algorithm. Alternatively, the AI model can be a combination of the above examples or any other AI model. The AI model used by the processor 1200 can be stored in the device determination model 1340.

[0136] The device determination model 1340 stored in the memory 1300 of the hub device 1000 can determine an operation execution device from among a plurality of devices 4000 registered according to the user account of the hub device 1000. The hub device 1000 can receive device information related to each of the plurality of devices 4000 from a voice assistant server 2000 by using the communication interface 1400. The device information can include at least one of the following: for example, identification information (e.g., device id information) of each of the plurality of devices 4000, the device type of each of the plurality of devices 4000, the function execution ability of each of the plurality of devices 4000, location information, or status information. The processor 1200 can determine, based on the device information, a device for performing an operation according to an intention from among the plurality of devices 4000 by using the data regarding the device determination model 1340 stored in the memory 1300.

[0137] In an embodiment, the processor 1200 can analyze a numerical value indicating the degree of relationship between an intention and a plurality of devices 4000 pre-registered by using the same user account as that of the hub device 1000 by using the device determination model 1340, and can determine, as the operation execution device, a device having the maximum value among the numerical values indicating the degree of relationship between the intention and the plurality of devices 4000.

[0138] Since the device determination model 1340 is configured to determine an operation execution device by using only a plurality of devices 4000 logged in and registered by using the same user account as that of the hub device 1000 as candidate devices, there is a technical effect that the amount of computation performed by the processor 1200 to determine the degree of relationship with an intention can be reduced to less than that of the processor 2200 of the voice assistant server 2000 (see Figure 3 ). In addition, due to the reduction in the amount of computation, the processing time required to determine the operation execution device can be reduced, and thus the response speed can be improved.

[0139] In an embodiment, the processor 1200 may obtain a device name from the text by using the first NLU model 1342, and may determine the operation execution device based on the device name by using the data of the device determination model 1340 stored in the memory 1300. In an embodiment, the processor 1200 may extract a general name related to the device and words or phrases about the device installation location from the text by using the first NLU model 1342, and may determine the operation execution device based on the extracted general name and the installation location of the device. For example, when the text converted from voice input is "Play the movie Avengers on the TV", the processor 1200 may parse the text in units of words or phrases by using the first NLU model 1342, and may identify the device name corresponding to "TV" by comparing the words or phrases with the pre-stored words or phrases. The processor 1200 may determine the second device 4200 among the multiple devices 4000 connected to the hub device 1000 as the operation execution device, where the second device 4200 is logged in with the same account as the user account of the hub device 1000 and is a TV.

[0140] The natural language generation (NLG) module 1320 may be used to provide a response message during the interaction between the hub device 1000 and the user. For example, the processor 1200 may generate a response message by using the NLG module 1320, such as "I will play the movie on the TV" or "I will lower the set temperature of the air conditioner by 2°C". The response message generated by the NLG module 1320 may be a message including text.

[0141] When there are multiple operation execution devices determined by the processor 1200 or there are multiple devices with a relationship degree similar to the intention, the NLG module 1320 may store data for generating a query message for determining a specific operation execution device. In an embodiment, the processor 1200 may generate a query message for selecting one operation execution device from multiple candidate devices by using the data about the NLG module 1320. The query message may be a message for requesting a response from the user about which one of the multiple candidate devices will be determined as the operation execution device.

[0142] The TTS module 1330 can convert a response message or a query message including the text generated by the NLG module 1320 into an audio signal. In an embodiment, the processor 1200 can control to convert a response message or a query message including text into an audio signal by using the data regarding the TTS module 1330 and send it through the speaker. In an embodiment, the hub device 1000 can receive a notification message indicating an operation execution result from the operation execution device. In this case, the notification information can include text, and the processor 1200 can convert the notification message including the text into an audio signal by using the data regarding the TTS module 1330. The processor 1200 can output the notification message converted into an audio signal through the speaker.

[0143] When the hub device 1000 obtains information about a new device connected to the hub device 1000, the hub device 1000 requests the voice assistant server 2000 to update the device determination model stored in the hub device 1000. In an embodiment, the hub device 1000 can analyze the text related to the new device based on the device information of the new device (e.g., the identification information of the new device and the type information of the new device) received from the IoT server 3000 by the user's voice input, and can send a query signal for requesting to update the device determination model pre-stored in the memory 1300 (see Figure 2 ) to the voice assistant server 2000 to determine the new device as an operation execution device. In this case, the hub device 1000 can send at least the user account information and the identification information of the hub device 1000 (e.g., the id information of the hub device 1000) together with the query signal to the voice assistant server 2000.

[0144] In another embodiment, the hub device 1000 can receive the device information of the new device (e.g., the identification information of the new device and the type information of the new device) from the new device. In this case, the hub device 1000 can send a query signal for requesting to update the device determination model pre-stored in the hub device 1000 based on the device information of the new device received from the new device.

[0145] In response to the request of the hub device 1000, the hub device 1000 receives the updated device determination model from the voice assistant server 2000 and stores the received updated device determination model. In an embodiment, the hub device 1000 can analyze the user voice input regarding the new device. As an analysis result, it can download the updated device determination model from the voice assistant server 2000 to determine the new device as an operation execution device, and can store the updated device determination model in the memory 1300 inside the hub device 1000 (see Figure 2) Among them. The operations of the request update device for determining the model and the device for receiving and storing the update for determining the model can be performed by separate software modules. Alternatively, these operations can be performed by the function determination model management module 1360.

[0146] The resource tracking module 1350 is a module that monitors the resource status of the elements of the hub device 1000 and the multiple devices 4000 (e.g., the processors 1200, 4120, 4220, and 4320 and the memories 1300, 4130, 4230, and 4330) and obtains resource status information. The resource status information is information indicating the usage status of the processors 1200, 4120, 4220, and 4320 and the memories 1300, 4130, 4230, and 4330 of the hub device 1000 and the multiple devices 4000. The resource status information may include, for example, at least one of the remaining capacity of the memories 1300, 4130, 4230, and 4330 of the hub device 1000 and the multiple devices 4000, the average RAM usage capacity, or the average processor occupancy information. In an embodiment, the hub device 1000 can periodically or in real time monitor at least one of the average processing speed of the processors 1200, 4120, 4220, and 4320, the remaining capacity of the memories 1300, 4130, 4230, and 4330, or the average remaining capacity of the RAM by using the resource tracking module 1350. The resource tracking module 1350 can provide the resource status information obtained through monitoring to the function determination model management module 1360.

[0147] The function determination model management module 1360 can determine whether the function determination model 1370 is stored and processed in the memories 1300, 4130, 4230, and 4330 of the hub device 1000 and the multiple devices 4000. When multiple function determination models 1370 are provided, the function determination model management module 1360 can determine whether the multiple function determination models 1370 are respectively stored and processed in the memories 1300, 4130, 4230, and 4330 of the hub device 1000 and the multiple devices. The function determination model management module 1360 can determine which one of the memories 1300, 4130, 4230, and 4330 of the hub device 1000 and the multiple devices 4000 will be used to store the function determination model 1370.

[0148] The function determination model management module 1360 can obtain from the voice assistant server 2000 (see Figure 1)The information of the received function determination model (e.g., the capacity information of the function determination model) is compared with the resource status information received from the resource tracking module 1350, and the information is analyzed. The function determination model management module 1360 can download the function determination model from the voice assistant server 2000 by using the comparison / analysis result, and can determine whether to store the downloaded function determination model in the memories 1300, 4130, 4230, and 4330 of the hub device 1000 and the multiple devices 4000. In an embodiment, the function determination model management module 1360 can determine whether to store the function determination model received from the voice assistant server 2000 corresponding to the new device 4300 (see Figure 1 ) in the memories 1300, 4130, 4230, and 4330. In an embodiment, the function determination model management module 1360 can analyze the remaining capacity and occupancy of the processors 1200, 4120, 4220, and 4320 and the memories 1300, 4130, 4230, and 4330 based on the resource status information obtained from the resource tracking module 1350 according to the usage patterns of the hub device 1000 and the multiple devices 4000. For example, when the user frequently plays music or video content through the hub device 1000, decoding occurs continuously, so the occupancy of the processor 1200 calculated is high and the remaining capacity of the RAM is low. In another embodiment, when the user only performs simple operations such as asking for weather information by using one of the multiple devices 4000, the occupancy of the processors 4120, 4220, and 4320 is not high and the remaining capacity of the memories 4130, 4230, and 4330 is large. The function determination model management module 1360 of the hub device 1000 can determine whether to store the function determination model in the memory 1300 of the hub device 1000 or the memories 4130, 4230, and 4330 of the multiple devices 4000 by comparing the resource status information of the processors 1200, 4120, 4220, and 4320 and the memories 1300, 4130, 4230, and 4330 with the information of the function determination model. The information of the function determination model can include at least one of the following: the capacity information of the function determination model, the RAM occupancy information for reading the function determination model, or the information of the simulation processing time required for the processors 1200, 4120, 4220, and 4320 to obtain operation information through detailed operations according to the functions and sequences of the detailed operations by reading the function determination model.

[0149] In an embodiment, the hub device 1000 may perform a simulation by using data on the function determination model management module 1360, based on information on the function determination model received from the voice assistant server 2000, to compare the remaining capacities of the memories 1300, 4130, 4230, and 4330 with the capacity information of the received function determination model. In an embodiment, the hub device 1000 may perform a simulation by using data on the function determination model management module 1360 to compare the average remaining RAM capacity connected to the processors 1200, 4120, 4220, and 4320 with the RAM capacity for processing the function determination model. In an embodiment, considering the average processing speed of the processor 1200, the hub device 1000 may simulate the processing time required to process the function determination model by using data on the function determination model management module 1360.

[0150] In an embodiment, when the difference between the remaining capacity of the memory 1300 and the capacity of the function determination model is less than a preset threshold, the function determination model management module 1360 may determine not to store the function determination model in the memory 1300. In an embodiment, when the simulated processing time of the function determination model is greater than a preset threshold time, the function determination model management module 1360 may determine not to store the function determination model in the memory 1300. The following will refer to Figure 7 A detailed method for determining whether to store the function determination model in the memory 1300 by the hub device 1000 by using data on the function determination model management module 1360 will be described in detail.

[0151] In another embodiment, the function determination model management module 1360 may analyze the characteristics of the function determination model corresponding to the new device 4300. The characteristics of the function determination model are analyzed based on the detailed information on the function to be performed by the device. The detailed information on the function may include the name of the function, the detailed description of the function, the resources required to execute the function (e.g., storage amount and processing speed), the applications required to execute the function, and the content type to which the function is applied. The function determination model management module 1360 obtains the detailed information on the function and analyzes the characteristics of the function determination model according to the pre-stored user voice assistant usage patterns, one or more pre-determined rules, and the type of sensitive information pre-input by the user.

[0152] The features of the function determination model include the minimum time required for a device to execute a function and output a result through the device or another peripheral device, and the required minimum time can be determined by a predetermined rule. For example, the hub device 1000 can store the following information: the default minimum time indicating the on / off function of the device is 0.1 ms, and the default minimum time for the function of searching and playing video content in the device is 1 s. Additionally, when the device that receives the user's voice input and the device that undergoes the function are different from each other, the hub device 1000 can store a rule indicating that 0.5 ms is added to the default minimum time.

[0153] Additionally, by considering the user's voice assistant usage pattern, the features of the function determination model include whether the function is similar to a function frequently used by the user. For example, the hub device 1000 can store usage pattern information indicating that the hub device 1000 plays music once every two days. In this case, by using the detailed information about the function included in the function determination model of the new device, it can be determined that the function determination model of the new device includes a function related to music playback, and this function is the same as the function the user executes once every two days.

[0154] Additionally, the features of the function determination model can include whether the user's sensitive information is used to execute the function, and the user's sensitive information can be determined according to a predetermined rule or according to the type of sensitive information pre-entered by the user. For example, when the function determination model of the new device has the function of obtaining schedule information and registering the schedule information in the calendar application, the hub device 1000 determines whether the schedule information is the user's sensitive information. The hub device 1000 can determine that the schedule information is the user's sensitive information according to the type of sensitive information pre-entered by the user.

[0155] The function determination model management module 1360 can determine whether to download the function determination model corresponding to the new device by analyzing the features of the function determination model. For example, as a result of analyzing the features of the function determination model, when the minimum time required to execute some functions included in the function determination model and output a result is equal to or less than a certain time, the function determination model management module 1360 can determine to download the function determination model to the hub device 1000 or the new device.

[0156] As another example, as a result of analyzing the features of the function determination model, when it is determined that some functions included in the function determination model are similar to the functions frequently used by the user, the function determination model management module 1360 can determine to download the function determination model to the hub device 1000 or the new device. As another example, as a result of analyzing the features of the function determination model, when it is determined that some functions included in the function determination model use the user's sensitive information, the function determination model management module 1360 can determine to download the function determination model to the hub device 1000 or the new device.

[0157] As another example, the function determination model management module 1360 can obtain information about the manufacturer or model name of a new device, and can determine whether to download a function determination model corresponding to the new device based on the information about the manufacturer or model name. The hub device 1000 can pre-store data on whether to download a function determination model for each manufacturer or model name. When it is determined by using the pre-stored data that the information about the manufacturer or model name of the new device is specified to download a function determination model, the hub device 1000 can determine to download the function determination model. Additionally, when the manufacturer information of the hub device 1000 and the manufacturer information of the new device are compared with each other and determined to be the same, the hub device 1000 can determine to download the function determination model corresponding to the new device.

[0158] In another embodiment, the function determination model management module 1360 can obtain information about the manufacturer or model name of a new device; and when downloading a function determination model corresponding to the new device based on the information about the manufacturer or model name, the function determination model management module 1360 can also download a data adapter module (not shown) together with the function determination model. The data adapter module (not shown) refers to a software (SW) module that converts the output data of the voice assistant server into data that can be processed by the IoT server to link the voice assistant server and the IoT server. In an embodiment, the data adapter module can determine the similarity between the data obtained by the device from the outside and the data stored inside the device, and can convert the data obtained from the outside into the data that is most similar to it. The conversion of the data can be performed based on a database including data pairs before and after conversion. In this case, the data adapter module determines the similarity between the data obtained from the outside and the data before conversion, and identifies the data that is most similar to the data obtained from the outside from the data before conversion. The data adapter module identifies the corresponding data after conversion by using the identified data before conversion, and outputs the identified data after conversion. In another embodiment, the data adapter module can convert the data obtained from the outside by using a data conversion model instead of a database. Alternatively, the data adapter module can use a database and a data conversion model.

[0159] The data that can be processed by the IoT server can be the same type of data as the data received by at least one of the multiple devices from the IoT server and used to perform a specific function. Additionally, the data adapter module can perform an operation of converting the output data of the IoT server into data that can be processed by the voice assistant server. The data that can be processed by the voice assistant server can be the same type of data as the information about at least one of the multiple devices received by the hub device 1000 from the voice assistant server.

[0160] For example, the output data of the voice assistant server includes information about detailed operations, and the output data of the IoT server includes device information. The device information includes at least one of identification information of the device, device type, function execution ability, location information, or status information. When the hub device 1000 downloads the data adapter module to the hub device 1000, the data adapter module (not shown) converts the output data of the hub device 1000 to link the hub device and the new device. Specifically, the data adapter module converts the output data of the function determination model into data that can be processed by the new device to link the function determination model downloaded to the hub device 1000 and the SW module (not shown) for executing the detailed operations stored in the new device. Additionally, when the hub device 1000 receives device information about the new device from the new device, the data adapter module converts the received device information into data that can be processed by the hub device 1000.

[0161] The data adapter module can be configured with the manufacturer or model name corresponding to the new device. In this case, the voice assistant server can store the data adapter module for each manufacturer or model name, and multiple data adapter modules can be stored. The hub device 1000 can request the data adapter module corresponding to the new device from the voice assistant server based on the manufacturer or model name information of the new device. Additionally, even without a request from the hub device 1000, the voice assistant server can send the data adapter module corresponding to the new device to the hub device 1000 based on the manufacturer or model name information of the new device obtained from the IoT server.

[0162] In an embodiment, when the function determination model management module 1360 determines to download the function determination model from the voice assistant server 2000 and store the function determination model in the memory 1300, the hub device 1000 can send a query to the voice assistant server 2000 for requesting the sending of the function determination model, can receive the function determination model from the voice assistant server 2000, and can store the function determination model in the memory 1300. In this case, the hub device 1000 can request the data adapter module corresponding to the new device from the voice assistant server based on the manufacturer or model name information of the new device. The data adapter module (not shown) converts the output data of the function determination model downloaded and stored in the hub device 1000 into data that can be processed by the new device.

[0163] In another embodiment, when the function determination model management module 1360 determines not to download the function determination model from the voice assistant server 2000, the hub device 1000 may receive access information of the function determination model pre-stored in the voice assistant server 2000, and may store the received access information in the function determination model information database 1362. The access information may include at least one of the following: identification information of the voice assistant server 2000 (see Figure 1 ), such as server id, location information, Internet Protocol (IP) address information, Media Access Control (MAC) address, application programming interface (API) information accessible to each function determination model in the voice assistant server 2000, the language used for each function determination model, or identification information of the new device.

[0164] In an embodiment, when registering a new device by using a user account, the hub device 1000 may receive information about the function determination model corresponding to the new device (e.g., capacity information) from the voice assistant server 2000 by using data about the function determination model management module 1360, may download the function determination model from the voice assistant server 2000 based on the received information about the function determination model, and may determine whether to store the function determination model in the memory 1300. When it is determined to store the function determination model in the memory 1300, the hub device 1000 may download the function determination model corresponding to the new device from the voice assistant server 2000, and may store the function determination model in the memory 1300. When it is determined not to store the function determination model, the hub device 1000 may store the access information of the function determination model corresponding to the new device in the function determination model information database 1362.

[0165] The hub device 1000 may determine the function of the hub device 1000 itself, and may store a first function determination model 1372 for performing operations according to the determined function and a second function determination model 1374 corresponding to at least one of the multiple devices 4000 (see Figure 1 ). The term "function determination model corresponding to the operation execution device" refers to a model for obtaining the following operation information: detailed operations for performing operations according to the determined function of the operation execution device and the relationships between the detailed operations. In an embodiment, the first function determination model 1372 and the second function determination model 1374 stored in the memory 1300 of the hub device 1000 may respectively correspond to multiple devices that are logged in by using the same account as the user account and connected to the hub device 1000 through a network.

[0166] For example, the first function determination model 1372 may be a model for obtaining the following operation information: according to the first device 4100 (see Figure 1) The detailed operations of the function execution operations and the relationships between the detailed operations. In an embodiment, the first function determination model 1372 may be, but is not limited to, a model for obtaining operation information according to the functions of the hub device 1000. Similarly, the second function determination model 1374 may be a model for obtaining operation information as follows: according to the second device 4200 (see Figure 1 ) The detailed operations of the function execution operations and the relationships between the detailed operations of the operation information.

[0167] The first function determination model 1372 and the second function determination model 1374 may respectively include a second NLU model 1372a and 1374a, wherein the second NLU models 1372a and 1374a are configured to analyze at least part of the text, and obtain operation information about the operations to be performed by the determined operation execution device based on the analysis results of at least part of the text. The first function determination model 1372 and the second function determination model 1374 may respectively include an action planning management model 1372b and 1374b, wherein the action planning management models 1372b and 1374b are configured to manage operation information related to the detailed operations of the device to generate the detailed operations to be performed by the device and the execution order of the detailed operations. The action planning management models 1372b and 1374b may manage operation information according to the device detailed operations of the device and the relationships between the detailed operations. The action planning management models 1372b and 1374b may plan the detailed operations to be performed by the device and the execution order of the detailed operations based on the analysis results of at least part of the text.

[0168] The processor 1200 may obtain operation information about the operations to be performed by the operation execution device by using the function determination model corresponding to the operation execution device stored in the memory 1300 (for example, the second NLU model 1374a of the second function determination model 1374 as the TV function determination model). The second NLU model 1374a, which is a module dedicated to a specific device (such as a TV), may be an AI model trained to obtain an intention related to the device, which corresponds to the operation execution device determined by the first NLU model 1342 and corresponds to the text. In addition, the second NLU model 1374a may be a model trained to determine device operations related to the user intention by interpreting the text. The operation may refer to at least one action performed by the device when the device executes a specified function. The operation may indicate at least one action performed by the device when the device executes an application.

[0169] In an embodiment, the processor 1200 may analyze text by using the second NLU model 1374a of the second function determination model 1374 corresponding to the determined operation execution device (e.g., TV). The processor 1200 may parse the text in units of morphemes, words, or phrases by using the second NLU model 1374a, may identify the meanings of the morphemes, words, or phrases parsed through syntactic or semantic analysis, and may determine the intent and parameters by matching the identified meanings with predefined words. The "parameters" used herein refer to variable information for determining the detailed operations of the operation execution device related to the intent. For example, when the text sent to the second function determination model 1374 is "Play the movie Avengers on the TV", the intent may be "Content playback", and the parameter may be "movie Avengers", i.e., information about the content to be played.

[0170] The processor 1200 may obtain at least one detailed operation operation information related to the intent and parameters by using the action planning management model 1374b of the second function determination model 1374. The action planning management model 1374b may manage operation information regarding the device detailed operations of the device and the relationships between the detailed operations. The processor 1200 may plan the detailed operations to be executed by the operation execution device (e.g., TV) and the execution order of the detailed operations based on the intent and parameters by using the action planning management model 1374b, and may obtain the operation information. The operation information may be information related to the detailed operations to be executed by the device and the execution order of the detailed operations. The operation information may include information related to the detailed operations to be executed by the device, the relationships between each detailed operation and another detailed operation, and the execution order of the detailed operations. The operation information may include, but is not limited to, the functions executed by the operation execution device to perform a specific operation, the execution order of the functions, the input values required to execute the functions, and the output values output as the execution results of the functions.

[0171] The processor 1200 may generate a control command for controlling the operation execution device based on the operation information. The control command refers to an instruction that can be read and executed by the operation execution device, so that the operation execution device executes the detailed operations included in the operation information. In this case, the processor 1200 must receive a control command conversion module for converting the operation information into a control command from the IoT server 3000 through the communication interface 1400. In an embodiment, the processor 1200 may send a query to the IoT server 3000 for requesting the sending of a control command conversion module, where the control command conversion module is used to convert the operation information about the operation execution device into a control command; may receive the control command conversion module from the IoT server 3000, and may store the control command conversion module in the memory 1300 through the control communication interface 1400. However, the present disclosure is not limited thereto, and the control command conversion module corresponding to the operation execution device may be pre-stored in the memory 1300.

[0172] The processor 1200 may control the communication interface 1400 to send the generated control command to the operation execution device.

[0173] The communication interface 1400 may perform data communication with the voice assistant server 2000, the IoT server 3000, and the plurality of devices 4000. The communication interface 1400 may perform data communication with the voice assistant server 2000, the IoT server 3000, and the plurality of devices 4000 by using at least one of data communication methods including wired LAN, wireless LAN, Wi-Fi, Bluetooth, Zigbee, Wi-Fi Direct (WFD), Infrared Data Association (IrDA), Bluetooth Low Energy (BLE), Near Field Communication (NFC), Wireless Broadband Internet (Wibro), Worldwide Interoperability for Microwave Access (WiMAX), Shared Wireless Access Protocol (SWAP), Wireless Gigabit Alliance (WiGig), or Radio Frequency (RF) communication.

[0174] In an embodiment, the processor 1200 may receive, through the communication interface 1400, information about a new device 4300 (see Figure 1 ) newly registered in the IoT server 3000 (see Figure 1 ) by logging in with the same account as the user account of the hub device 1000. The processor 1200 may receive information about the new device 4300 from the IoT server 3000 by using the communication interface 1400, but the present disclosure is not limited thereto. The IoT server 3000 may send information to the voice assistant server 2000 (see Figure 1)Send device information and user account information about the newly registered new device 4300 by using the user account, and the processor 1200 of the hub device 1000 can control the communication interface 1400 to receive the device information of the new device 4300 from the voice assistant server 2000.

[0175] The processor 1200 can request the voice assistant server 2000 to update the device determination model 1340 stored in the memory 1300 through the communication interface 1400, and can receive the updated device determination model 1340 from the voice assistant server 2000. When a new device 4300 is newly registered in the IoT server 3000 by using the user account, the voice assistant server 2000 can update the device determination model 2330 (see Figure 1 ) and the function determination model 2368 (see Figure 1 ). In this case, the processor 1200 can send a query to the voice assistant server 2000 for asking whether there is an updated device determination model 2330 for the device determination of the new device 4300, and when there is an updated device determination model 2330, the processor 1200 can control the communication interface 1400 to send a signal for requesting to send the updated device determination model 2330 to the hub device 1000.

[0176] The processor 1200 can analyze the text related to the new device 4300 from the voice assistant server 2000 through the communication interface 1400, can receive the updated device determination model 2330 to determine the new device 4300 as an operation execution device, and can store the received device determination model 2330 in the memory 1300.

[0177] The processor 1200 can send a query to the voice assistant server 2000 through the communication interface 1400 for requesting information about the function determination model corresponding to the new device 4300. The information about the function determination model corresponding to the new device 4300 can include at least one of, for example, the capacity information or access information of the function determination model, so as to obtain information related to the detailed operations performed by the new device 4300, the relationship between each detailed operation and another detailed operation, and the execution order of the detailed operations. Based on whether the function determination model management module 1360 determines that there is a function determination model corresponding to the new device 4300, the processor 1200 can control the communication interface 1400 to download the function determination model from the voice assistant server 2000 or only receive the access information of the function determination model.

[0178] In Figure 1 and Figure 2In the case where the new device 4300 logs in and uses the user account registration by using the account information of the same user as that of the hub device 1000, or when new functions are added to the existing device, the hub device 1000 may download the updated device determination model 2330 from the voice assistant server 2000 (see Figure 1 ), and may store the updated device determination model 2330 in the memory 1300, and may determine whether to download the function determination model corresponding to the new device 4300 or the updated function determination model corresponding to the existing device based on the resource status information in the hub device 1000, and store the downloaded function determination model in the memory 1300. Generally, the processing speed when the hub device 1000 determines the operation execution device and obtains the operation information about the operation execution device may be higher than the processing speed when the voice assistant server 2000 executes the processing, and the network usage fee for accessing the voice assistant server 2000 may be reduced. However, resource problems may occur when all the updated function determination models are stored in the memory 1300 of the hub device 1000 itself. Therefore, according to the embodiment, since the updated device determination model is downloaded from the voice assistant server 2000 and stored in the memory 1300, but whether to download the updated function determination model and the function determination model corresponding to the new device 4300 is determined after checking the resources of the hub device 1000 itself, the resource problem can be solved and the processing speed of the voice command can be improved.

[0179] Figure 3 is a block diagram of the voice assistant server 2000 according to the embodiment.

[0180] The voice assistant server 2000 is a server that receives the text converted from the user's voice input from the hub device 1000, determines the operation execution device based on the received text, and obtains the operation information by using the function determination model corresponding to the operation execution device.

[0181] Referring to Figure 3 , the voice assistant server 2000 may at least include a communication interface 2100, a processor 2200, and a memory 2300.

[0182] The communication interface 2100 of the voice assistant server 2000 may receive the device information including at least one of the following from the IoT server 3000 (see Figure 1 ) by performing data communication with the IoT server 3000: a plurality of devices 4000 (see Figure 1identification information (e.g., device ID information) of each of the devices in (), device types of each of the multiple devices 4000, function execution capabilities of each of the multiple devices 4000, location information, or status information. In an embodiment, the voice assistant server 2000 may receive, via the communication interface 2100, user account information used to register a new device 4300 and device information about the new device 4300 from the IoT server 3000. The voice assistant server 2000 may receive user account information from the hub device 1000 via the communication interface 2100, and may send device information about the multiple devices 4000 registered according to the received user account information to the hub device 1000. In an embodiment, the voice assistant server 2000 may send the updated device determination model 2330 to the hub device 1000 via the communication interface 2100. In an embodiment, the voice assistant server 2000 may send, via the communication interface 2100, information about the function determination model 2368 corresponding to the new device 4300 to the hub device 1000, or may send the function determination model 2368 itself to the hub device 1000.

[0183] In another embodiment, when the function determination model management module 1360 of the hub device 1000 (see Figure 2 ) determines that the function determination model 2368 corresponding to the new device 4300 is to be sent to one of the multiple devices 4000 other than the hub device 1000, the voice assistant server 2000 may receive, via the communication interface 2100, a query including identification information about the determined device and a signal indicating to send the function determination model from the hub device 1000, and may send the function determination model 2368 to the determined device in response to receiving the query.

[0184] The processor 2200 and the memory 2300 of the voice assistant server 2000 may perform functions the same as or similar to those of the processor 1200 (see Figure 2 ) and the memory 1300 (see Figure 2 ) of the hub device 1000 (see Figure 2 ). Therefore, the same description of the processor 1200 and the memory 1300 of the hub device 1000 is not provided for the processor 2200 and the memory 2300 of the voice assistant server 2000.

[0185] In the memory 2300 of the voice assistant server 2000, data about the ASR module 2310, data about the NLG module 2320, data about the device determination model 2330, data about the voice assistant model update module 2340, data related to the on-device model update module 2350, and data corresponding to each of the multiple function determination models 2360 can be stored. The memory 2300 of the voice assistant server 2000 can store multiple function determination models 2360 corresponding to multiple devices related to multiple different user accounts, rather than the multiple function determination models 1370 stored in the memory 1300 of the hub device 1000 (see Figure 2 ). Additionally, compared with the multiple function determination models 1370 stored in the memory 1300 of the hub device 1000, multiple function determination models 2360 for more types of devices can be stored in the memory 2300 of the voice assistant server 2000. The total capacity of the multiple function determination models 2360 stored in the memory 2300 of the voice assistant server 2000 can be greater than the capacity of the multiple function determination models 1370 stored in the memory 1300 of the hub device 1000.

[0186] When receiving at least a portion of text from the hub device 1000, the communication interface 2100 of the voice assistant server 2000 can send the received at least a portion of text to the processor 2200, and the processor 2200 can analyze the at least a portion of text by using the first NLU model 2332 stored in the memory 2300. The processor 2200 can determine the operation execution device related to the at least a portion of text based on the analysis result by using the device determination model 2330 stored in the memory 2300. The processor 2200 can select a function determination model corresponding to the operation execution device from the multiple function determination models stored in the memory 2300, and can obtain operation information about the detailed operations for the operation execution device to perform functions and the relationships between the detailed operations by using the selected function determination model.

[0187] For example, when it is determined that the operation execution device is the first device 4100 serving as an air conditioner, the processor 2200 can analyze the at least a portion of text by using the second NLU model 2362a of the function determination model 2362 corresponding to the air conditioner, and can obtain operation information by planning the detailed operations to be performed by the device and the execution order of the detailed operations by using the action planning management model 2362b.

[0188] For example, when the voice assistant server 2000 receives text corresponding to "raise the temperature by 1°C" from the hub device 1000, the voice assistant server 2000 determines the air conditioner as the operation execution device through the device determination model 2330. Subsequently, the voice assistant server 2000 can select the function determination model 2362 corresponding to the air conditioner from among the multiple function determination models 2360, can analyze the text by using data related to the second NLU model 2362a of the selected function determination model 2362, and can obtain operation information for performing the temperature control operation based on the text analysis result by using data related to the action planning management model 2362b.

[0189] For example, when the voice assistant server 2000 receives text corresponding to "change the channel" from the hub device 1000, the voice assistant server 2000 determines the TV as the operation execution device through the device determination model 2330. Subsequently, the voice assistant server 2000 can select the function determination model 2364 corresponding to the TV from among the multiple function determination models 2360, can analyze the text by using data regarding the second NLU model 2364a of the selected function determination model 2364, and can obtain operation information for performing the temperature control operation based on the text analysis result by using data regarding the action planning management model 2364b.

[0190] The voice assistant server 2000 can update the voice assistant model by using the voice assistant model update module 2340 stored in the memory 2300. When a new device 4300 is newly registered in the IoT server 3000 by using a user account (see Figure 1 ) or a new function is added to an existing device, the voice assistant server 2000 can receive the user account and the device information of the newly registered new device 4300 by using the user account from the IoT server 3000 through the communication interface 2100. The voice assistant server 2000 can update the device determination model 2330 and the function determination model 2368 corresponding to the new device 4300 by using the voice assistant model update module 2340.

[0191] For example, when the function determination model 2364 corresponding to the second device 4200 is updated, for example, when the second device 4200 (see Figure 1When the function of () is added or changed, the voice assistant model update module 2340 can update the device determination model 2330 to a new model through learning or the like, so that the device determination model 2330 determines the second device 4200 as an operation execution device related to the updated function by interpreting the updated function. In an embodiment, the voice assistant model update module 2340 can update the first NLU model 2332 to a new model through learning or the like, so that the first NLU model 2332 of the device determination model 2330 interprets the text related to the newly updated function.

[0192] For example, when a new device 4300 is registered in the IoT server 3000 by using a user account, the voice assistant model update module 2340 can receive the identification information of the new device 4300 and the function determination model 2368 corresponding to the new device 4300 from the IoT server 3000, and can also store the received function determination model 2368 in the function determination models 2362, 2364, and 2366 stored in the memory 2300. The voice assistant model update module 2340 can update the device determination model 2330 to a new model through learning or the like, so that the device determination model 2330 interprets the intention from the text, and as a result of the interpretation, determines the new device 4300 as an operation execution device related to the intention. In an embodiment, the voice assistant model update module 2340 can update the first NLU model 2332 to a new model through learning or the like, so that the first NLU model 2332 of the device determination model 2330 interprets the text related to the new device 4300.

[0193] The voice assistant server 2000 can train the voice assistant model according to the user account by using the on-device model update module 2350, and can determine whether to send at least some of the voice assistant models updated through learning to the hub device 1000 (see Figure 2 ). In another embodiment, the voice assistant server 2000 can determine to send at least some of the updated voice assistant models to at least one of the plurality of devices 4000 by using the on-device model update module 2350.

[0194] The on-device model update module 2350 can include an on-device model information database 2352 that stores the configuration information of the voice assistant model according to the user account. The on-device model information database 2352 can store at least one of the following: for example, user account information, identification information about a plurality of devices registered according to the user account, identification information of the function determination model corresponding to each of the plurality of devices, version information of the function determination model, version information of the device determination model, or device type information that can be determined as an operation execution device by the device determination model.

[0195] In an embodiment, the voice assistant server 2000 may determine to send the device determination model 2330 to be updated to the hub device 1000 by using the on-device model update module 2350 (see Figure 2 ). In an embodiment, the voice assistant server 2000 may send the updated device determination model 2330 to the hub device 1000 through the communication interface 2100. In another embodiment, the voice assistant server 2000 may send the updated device determination model 2330 to at least one of the multiple devices 4000 through the communication interface 2100.

[0196] In another embodiment, together with the information about the new device connected to the hub device 1000, the voice assistant server 2000 receives a query signal from the hub device 1000 for updating the device determination model pre-stored in the hub device 1000. In this case, together with the query signal, the voice assistant server 2000 may also receive at least the user account information and the identification information of the hub device 1000 (e.g., the id information of the hub device 1000) from the hub device 1000. The on-device model update module 2350 of the voice assistant server 2000 may update the device determination model pre-stored in the hub device 1000 by at least using the received account information and the information about the new device received from the hub device 1000.

[0197] The update of the device determination model is similar to the update of the device determination model 2330 of the voice assistant server 2000. That is, the on-device model update module 2350 may update the device determination model of the hub device 1000 to a new model by learning, etc., so that the device determination model of the hub device 1000 interprets the intention from the text, and as a result of the interpretation, determines the new device 4300 as the operation execution device related to the intention. In an embodiment, the on-device model update module 2350 may update the first NLU model to a new model by learning, etc., so that the first NLU model of the device determination model of the hub device 10 reproduces the text related to the new device 4300. The update of the module will be described in more detail below with reference to ​ and ​ .

[0198] In an embodiment, the voice assistant server 2000 may receive a signal from the hub device 1000 for requesting to send the function determination model 2368 corresponding to the new device 4300. When receiving the signal for requesting to send the function determination model 2368 corresponding to the new device 4300, the voice assistant server 2000 may send the function determination model 2368 corresponding to the new device 4300 to the hub device 1000 by using the on-device model update module 2350.

[0199] The voice assistant server 2000 may include a data adapter module (not shown). The data adapter module (not shown) refers to a software (SW) module that converts the output data of the voice assistant server 2000 into data that can be processed by the IoT server 3000 to link the voice assistant server 2000 and the IoT server 3000. The voice assistant server 2000 may store data adapter modules for each manufacturer or model name, and may include multiple data adapter modules. The voice assistant server 2000 may send the data adapter module corresponding to the new device to the hub device 1000 in response to a request from the hub device 1000. In an embodiment, even when there is no request from the hub device 1000, the voice assistant server 2000 may send the data adapter module corresponding to the new device to the hub device 1000 based on the manufacturer or model name information of the new device obtained from the IoT server 3000.

[0200] In another embodiment, the voice assistant server 2000 may receive from the hub device 1000 a query for requesting to send a function determination model 2368 corresponding to the new device 4300 to at least one of the multiple devices 4000. In this case, the voice assistant server 2000 may receive identification information (e.g., device id) about one of the multiple devices 4000 together with the query. The voice assistant server 2000 may send the function determination model 2368 corresponding to the new device to the device determined from the multiple devices 4000 according to the received device information by using the on-device model update module 2350.

[0201] ​ It is a block diagram of the IoT server 3000 according to an embodiment.

[0202] The IoT server 3000 is a server that obtains, stores, and manages device information about each of the multiple devices 4000 (see ​ ). The IoT server 3000 may obtain, determine, or generate a control command for controlling the device by using the stored device information. Although the IoT server 3000 is implemented as an independent hardware device separated from the voice assistant server 2000 in ​ , the present disclosure is not limited thereto. In an embodiment, the IoT server 3000 may be an element in the voice assistant server 2000 (see ​ ), or may be a server designed to be classified as software.

[0203] Refer to ​ , the IoT server 3000 may at least include a communication interface 3100, a processor 3200, and a memory 3300.

[0204] The IoT server 3000 can be connected to the voice assistant server 2000 or the operation execution device via a network through the communication interface 3100, and can receive or send data. The IoT server 3000 can, under the control of the processor 3200, send the data stored in the memory 3300 to the voice assistant server 2000 or the operation execution device through the communication interface 3100. Additionally, the IoT server 3000 can, under the control of the processor 3200, receive data from the voice assistant server 2000 or the operation execution device through the communication interface 3100.

[0205] In an embodiment, the communication interface 3100 can receive device information including at least one of the following from each of the multiple devices 4000 (see ​ ): device identification information (e.g., device id information), function execution ability information, location information, or status information. In an embodiment, the communication interface 3100 can receive user account information from each of the multiple devices 4000. Additionally, the communication interface 3100 can receive information about power on / off or the operations being performed from the multiple devices 4000. The communication interface 3100 can provide the received device information to the memory 3300.

[0206] In an embodiment, the communication interface 3100 can receive the user account information used to log in to the new device 4300 and device information about the new device 4300 from the new device 4300 (see ​ ).

[0207] In an embodiment, the communication interface 3100 can, under the control of the processor 3200, send the user account information of the new device 4300 and the device information of the new device 4300 registered by using the user account to the voice assistant server 2000 (see ​ ).

[0208] The processor 3200 can store the device information related to each of the multiple devices 4000 received through the communication interface 3100 in the memory 3300. In an embodiment, the memory 3300 can classify the device information according to the user account information received from the multiple devices 4000, and can store the classified device information in the form of a lookup table (LUT).

[0209] The processor 3200 may store, in the memory 3300, device information about a new device received through the communication interface 3100 and user account information for logging in to the new device 4300. In an embodiment, the processor 3200 may store the device information of the new device 4300 (e.g., at least one of identification information (e.g., id information), function execution capability information, location information, or status information of the new device 4300) in the user account information corresponding to the new device 4300 in the memory 3300 in the form of a lookup table.

[0210] In an embodiment, the communication interface 3100 may receive, from the voice assistant server 2000, a query for requesting user account information and device information about a plurality of devices 4000 pre-registered by using the user account information. In response to receiving the query, the processor 3200 may obtain, from the memory 3300, the device information about the plurality of devices 4000 pre-registered by using the user account in response to receiving the query, and may control the communication interface 3100 to send the obtained device information to the voice assistant server 2000.

[0211] In an embodiment, the communication interface 3100 may receive, from the voice assistant server 2000, a query for requesting device information about a new device 4300 newly registered by using the user account information. In this case, in response to receiving the query, the processor 3200 may control the communication interface 3100 to send the device information about the new device 4300 newly registered by using the user account to the voice assistant server 2000.

[0212] The processor 3200 may control the communication interface 3100 to send a control command to an operation execution device determined to execute an operation based on the operation information received from the voice assistant server 2000.

[0213] The IoT server 3000 may receive, through the communication interface 3100, an operation execution result according to the control command from the operation execution device.

[0214] ​ is a block diagram of a plurality of devices 4000 according to an embodiment.

[0215] The plurality of devices 4000 may be devices controlled by the hub device 1000 (see ​ ) or the IoT server 3000 (see ​ ). In an embodiment, the plurality of devices 4000 may be actuator devices that perform operations based on control commands received from the hub device 1000 or the IoT server 3000.

[0216] Reference ​, a plurality of devices 4000 may include a first device 4100, a second device 4200, and a new device 4300. However, the present disclosure is not limited thereto. In an embodiment, the plurality of devices 4000 may only include the pre-registered first device 4100 and second device 4200, and may not include the new device 4300.

[0217] The new device 4300 may be newly acquired by a user of the hub device 1000 and registered in the IoT server 3000 (see ​ ) by logging in with the same user account as other pre-registered devices 4100 and 4200.

[0218] Although in ​ , the first device 4100 is an air conditioner, the second device 4200 is a TV, and the new device 4300 is an air purifier, the present disclosure is not limited thereto. The plurality of devices 4000 is not limited to ​ the device types shown.

[0219] Although in ​ , each of the first device 4100, the second device 4200, and the new device 4300 only includes a processor, a memory, and a communication interface, this is for ease of explanation. In ​ , the elements required for each of the plurality of devices 4000 to perform operations based on a control command are not shown in ​ .

[0220] Some of the plurality of devices 4000 may store a function determination model. In ​ , the first device 4100 may include a communication interface 4110, a processor 4120, and a memory 4130, and the function determination model 4132 may be stored in the memory 4130. The function determination model 4132 stored in the first device 4100 may be a model for obtaining operation information regarding detailed operations performed by the first device 4100 and the relationships between the detailed operations. The function determination model 4132 may include an NLU model 4134, where the NLU model 4134 is configured to analyze at least part of the text received from the hub device 1000 or the IoT server 3000, and obtain operation information regarding the operations to be performed by the first device 4100 based on the analysis result of at least part of the text. The function determination model 4132 may include an action planning management model 4136, where the action planning management model 4136 is configured to manage operation information related to the detailed operations of the device to generate the detailed operations to be performed by the first device 4100 and the execution order of the detailed operations. The action planning management model 4136 may plan the detailed operations to be performed by the first device 4100 and the execution order of the detailed operations based on the analysis result of at least part of the text.

[0221] The second device 4200 may include a communication interface 4210, a processor 4220, and a memory 4230. The new device 4300 may include a communication interface 4310, a processor 4320, and a memory 4330. Different from the first device 4100, each of the second device 4200 and the new device 4300 does not store a function determination model. Each of the second device 4200 and the new device 4300 does not receive at least partial text from the hub device 1000 (see ​ ) or the IoT server 3000 (see ​ ). Each of the second device 4200 and the new device 4300 may receive a control command from the hub device 1000 or the IoT server 3000, and may perform an operation based on the received control command.

[0222] However, for the sake of easy explanation, the present disclosure is not limited thereto. In an embodiment, the new device 4300 itself may include a function determination model for obtaining operation information regarding detailed operations performed by the new device 4300, relationships between the detailed operations, and an execution order of the detailed operations.

[0223] The plurality of devices 4000 may send user account information and device information to the IoT server 3000 by using the communication interfaces 4110, 4210, and 4310. The device information may include at least one of the following: for example, identification information (e.g., device ID information) of each of the plurality of devices 4000, device types of each of the plurality of devices 4000, function execution capabilities of each of the plurality of devices 4000, location information, or status information.

[0224] In an embodiment, at least one of the plurality of devices 4000 may include an updated function determination model for adding, changing, or removing functions. For example, when a "no-wind function (new function)" is added to the first device 4100, the function determination model 4132 stored in the memory 4130 of the first device 4100 may be updated. When the function determination model 4132 of the first device 4100 is updated, the first device 4100 may send information regarding the updated function determination model 4132 to the IoT server 3000 by using the communication interface 4110.

[0225] The new device 4300 may send the user account information of the user logged in to the new device 4300 and device information to the IoT server 3000 through the communication interface 4310. In an embodiment, the new device 4300 may send information regarding whether the new device 4300 itself includes a function determination model to the IoT server 3000 by using the communication interface 4310.

[0226] In an embodiment, multiple devices 4000 can send user account information, information on whether each of the multiple devices 4000 itself stores a device determination model, and information on whether each of the multiple devices 4000 itself stores a function determination model to the hub device 1000 (see ​ ) by using the communication interfaces 4110, 4210, and 4310.

[0227] ​ FIG. is a flowchart of a method for the hub device 1000 to receive and store at least some voice assistant models according to an embodiment, which is executed by the hub device 1000.

[0228] In operation S610, the hub device 1000 receives information of a new device connected to the hub device 1000. In an embodiment, the hub device 1000 can receive device information of a new device newly registered in the IoT server 3000 (see ​ ) by logging in with the same user account as the hub device 1000 and using the user account. The new device can be a device newly obtained by the user of the hub device 1000 through purchase or transfer of ownership and registered in the IoT server 3000 by logging in with the same user account as the hub device 1000. The device information of the new device received by the hub device 1000 can include at least one of the following: for example, identification information of the new device (e.g., device id information), device type of the new device, function execution ability information of the new device, or location information. In an embodiment, the hub device 1000 can receive status information on the power on / off or currently executing operation of the new device.

[0229] The hub device 1000 can receive the device information of the new device from the IoT server 3000. However, the present disclosure is not limited thereto, and the hub device 1000 can receive the device information of the new device from the voice assistant server 2000 (see ​ ).

[0230] In operation S620, the hub device 1000 requests the voice assistant server 2000 to update the device determination model stored in the hub device 1000. In an embodiment, the hub device 1000 can send a request for update to the memory 1300 (see ​) The device in it determines that the query signal of the model is sent to the voice assistant server 2000, so that the hub device 1000 analyzes the text related to the new device according to the user's voice input based on the device information of the new device received in operation S610, and determines the new device as the operation execution device. In this case, the hub device 1000 may send the user account information of the new device and the identification information of the hub device 1000 (for example, the id information of the hub device 1000) to the voice assistant server 2000 together with the query signal.

[0231] In operation S630, the hub device 1000 receives and stores the updated device determination model from the voice assistant server 2000. In an embodiment, the hub device 1000 may analyze the user voice input related to the new device. As an analysis result, it may download the updated device determination model from the voice assistant server 2000 to determine the new device as the operation execution device, and may store the updated device determination model in the memory 1300 inside the hub device 1000 (see ​ ).

[0232] In operation S640, the hub device 1000 requests information about the function determination model corresponding to the new device from the voice assistant server 2000, and receives information about the function determination model corresponding to the new device from the voice assistant server 2000. In an embodiment, when the hub device 1000 requests information about the function determination model corresponding to the new device, the hub device 1000 may send the identification information of the hub device 1000 (for example, the device id) and the information of the new device and the user account to the voice assistant server 2000.

[0233] The function determination model corresponding to the new device is a model for obtaining information related to the detailed operations to be performed by the new device, the relationship between each detailed operation and another detailed operation, and the execution order of the detailed operations when the new device is determined as the operation execution device. The information of the function determination model may include at least one of the following: for example, the capacity information of the file constituting the function determination model corresponding to the new device, the RAM occupancy information for reading the function determination model, or the information of the simulation processing time required for the processor 1200 of the hub device 1000 (see ​ ) to obtain operation information according to the detailed operations and the order of the detailed operations of the function by reading the function determination model.

[0234] In operation S650, the hub device 1000 determines whether to store the function determination model corresponding to the new device in the hub device 1000 based on the received information about the function determination model corresponding to the new device. In an embodiment, the hub device 1000 may monitor the processor 1200 (see ​) and the resource status of the memory 1300 (see ​ ), and by comparing and analyzing the resource status information and the information of the function determination model corresponding to the new device, determine whether to store the function determination model corresponding to the new device in the memory 1300. The resource status information, as the information indicating the usage status of the processors 1200, 4120, 4220, and 4320 and the memories 1300, 4130, 4230, and 4330 of the hub device 1000 and the multiple devices 4000, may include at least one of the following: For example, the remaining capacity of the memories 1300, 4130, 4230, and 4330 of the hub device 1000 and the multiple devices 4000, the average RAM usage capacity, or the average processor occupancy information.

[0235] In an embodiment, the hub device 1000 may determine whether to download the function determination model corresponding to the new device from the voice assistant server 2000 and store the downloaded function determination model in the memory 1300 by monitoring at least one of the average processing speed of the processor 1200, the remaining capacity of the memory 1300, or the average remaining RAM capacity in real time or periodically, and comparing the monitoring result information with the information of the function determination model corresponding to the new device.

[0236] In another embodiment, the hub device 1000 may monitor the resource status information of each of the multiple devices 4000 (see ​ ), and may obtain the resource status information of each of the multiple devices 4000. The resource status information of the multiple devices 4000, as the information related to the usage status of the processors 4120, 4220, and 4320 (see ​ ) and the memories 1300, 4130, 4230, and 4330 (see ​ ), may include at least one of the remaining capacity of the memories 4130, 4230, and 4330, the average RAM usage capacity, or the average processor occupancy information. The hub device 1000 may determine whether to store the function determination model corresponding to the new device in at least one of the multiple devices 4000 based on the resource status information. The hub device 1000 may compare and analyze the information of the function determination model and the resource status information of each of the multiple devices 4000, and may determine which one of the memories 4130, 4230, and 4330 of the multiple devices 4000 is to be used to store the function determination model corresponding to the new device based on the comparison / analysis result.

[0237] The following will refer to ​ to describe in detail the specific process performed by the hub device 1000 in operation S650 to determine whether to store the function determination model corresponding to the new device.

[0238] When it is determined that the function determination model corresponding to the new device is to be stored (operation S660), the hub device 1000 receives the function determination model corresponding to the new device from the voice assistant server 2000 and stores the function determination model in the memory 1300 (see ​ ). Since the function determination model corresponding to the new device is stored in the memory 1300 of the hub device 1000, when the new device is determined as the operation execution device, the hub device 1000 can obtain information about the detailed operations of the new device, the relationships between the detailed operations, and the execution order of the detailed operations even without accessing the voice assistant server 2000. Therefore, the time consumed for accessing the voice assistant server 2000 and the network usage cost can be avoided, and the processing speed can be improved. In addition, when the network is unavailable (for example, when the wireless communication network or Wi-Fi communication is unavailable), the hub device 1000 itself can obtain the operation information about the new device.

[0239] Although the function determination model corresponding to the new device is stored in the memory 1300 of the hub device 1000 in operation S660, the present disclosure is not limited thereto. When in operation S650 the hub device 1000 determines to store the function determination model corresponding to the new device in at least one of the plurality of devices 4000 based on the resource status information of each of the plurality of devices 4000, in operation S660, the hub device 1000 can receive the function determination model from the voice assistant server 2000 and can send the function determination model to the determined device.

[0240] In another embodiment, the hub device 1000 can send the device information (for example, device id) of the device determined to store the function determination model corresponding to the new device to the voice assistant server 2000. In this case, the voice assistant server 2000 can send the determined function determination model corresponding to the new device based on the device information.

[0241] When it is determined not to store the function determination model corresponding to the new device (operation S670), the hub device 1000 receives the access information of the function determination model corresponding to the new device from the voice assistant server 2000. The access information may include at least one of the following: for example, the identification information (for example, server id) of the voice assistant server 2000 storing the function determination model corresponding to the new device, location information, IP address information, MAC address, the application programming interface (API) information through which the function determination model in the voice assistant server 2000 can be accessed, the language used for the function determination model, or the identification information of the device. The hub device 1000 can store the received access information in the function determination model information database 136 in the memory 1300 (see ​) Since the access information is stored in the memory 1300, even when the function determination model corresponding to the new device itself is not stored, the hub device 1000 can access the voice assistant server 2000 by using the access information, and can obtain information about the detailed operations of the new device, the relationships between the detailed operations, and the execution order of the detailed operations by using the function determination model corresponding to the new device stored in the voice assistant server 2000.

[0242] ​ is a flowchart of a method for determining whether to store a function determination model by the hub device 1000 according to an embodiment, based on information of the function determination model received from the voice assistant server 2000. ​ is ​ a detailed flowchart of operation S650 in ​ which can be executed after operation S640 in ​ and operations S710, S712, and S714 in

[0243] In operation S710, the hub device 1000 identifies the remaining capacity of the flash memory in the hub device 1000. The flash memory may be a storage medium for storing the updated device determination model and at least one function determination model in the hub device 1000. In an embodiment, the hub device 1000 may include a hard disk type, a solid state drive (SSD), or a card type external memory (e.g., SD or XD memory) as the storage medium instead of the flash memory type.

[0244] In operation S712, the hub device 1000 identifies the average remaining RAM capacity. The average remaining RAM capacity indicates the average available RAM capacity, excluding the RAM capacity used by at least one application for providing services executed by the hub device 1000 or at least one application for receiving push notifications executed in the background.

[0245] In operation S714, the hub device 1000 identifies the average processing speed. The hub device 1000 may identify the average processing speed of the processor 1200 (see ​ ). The average processing speed indicates the average computing speed at which the processor 1200 executes functions such as executing the functions of at least one application, converting the user's voice input into text by executing ASR, or determining the operation execution device related to the text by using the device determination model.

[0246] The operations S710, S712, and S714 for monitoring the resource status of the hub device 1000 and obtaining the resource status information may be performed by the processor 1200 by using the resource tracking module 1350 stored in the memory 1300 (see ​) is performed. Operations S710, S712, and S714 may be performed in real time or periodically.

[0247] Although operations S710, S712, and S714 may be performed simultaneously, the present disclosure is not limited thereto.

[0248] Operations S710, S712, and S714 may not all be performed, and at least one of operations S710, S712, and S714 may be performed. When one of operations S710, S712, and S714 is not performed, the operations after that operation may also not be performed. For example, when operation S710 is not performed, operation S720 after operation S710 may not be performed.

[0249] Although operations S710, S712, and S714 respectively identify the remaining capacity of the internal flash memory of the hub device 1000, the average remaining capacity of the RAM of the hub device 1000, and the average processing speed of the hub device 1000, the present disclosure is not limited thereto. In an embodiment, operations S710, S712, and S714 may respectively monitor the remaining capacity of the flash memory, the average remaining capacity of the RAM, and the average processing speed of a plurality of devices 4000 (see ​ ) other than the hub device 1000.

[0250] In operation S720, the hub device 1000 compares the remaining capacity of the flash memory with the information received regarding the function determination model. The information regarding the function determination model received in operation S640 may include at least one of the following: the capacity information of the files constituting the function determination model, the RAM occupancy information for reading the function determination model, or the information on the simulation processing time required for the processor 1200 of the hub device 1000 (see ​ ) to obtain operation information based on the detailed operations of the function and the order of the detailed operations by reading the function determination model. The hub device 1,000 may compare the remaining capacity of the flash memory with the capacity information of the files constituting the function determination model. In an embodiment, the hub device 1000 may compare the remaining capacity of the flash memory of each of the plurality of devices 4000 with the capacity information of the function determination model.

[0251] In operation S722, the hub device 1000 simulates the operation of the function determination model in consideration of the remaining RAM capacity. In an embodiment, the hub device 1000 may compare the average remaining RAM capacity identified in operation S712 with the RAM occupancy information in the function determination model information received in operation S640, and simulate the RAM occupancy capacity used by the processor 1200 to read and execute the instructions written in a programming language included in the function determination model. In an embodiment, the hub device 1000 may simulate the operation of the function determination model in consideration of the remaining RAM capacity of each of the plurality of devices 4000.

[0252] In operation S724, the hub device 1000 simulates the operation of the function determination model in consideration of the average processing speed. In one embodiment, the hub device 1000 may compare the average processing speed identified in operation S714 with the simulated processing time information in the function determination model information received in operation S640, and simulate the processing speed for reading the function determination model and obtaining operation information by following the detailed operations of the read function and the order of the detailed operations. In an embodiment, the hub device 1000 may simulate the operation of the function determination model in consideration of the average processing speed of each of the processors 4120, 4220, and 4320 of the plurality of devices 4000.

[0253] In operation S730, the hub device 1000 determines whether the difference between the remaining memory capacity and the capacity of the function determination model is greater than a preset remaining memory capacity threshold. In an embodiment, the hub device 1000 may calculate the difference between the remaining flash memory capacity and the capacity information of the files constituting the function determination model. The capacity information of the files constituting the function determination model may be obtained from the function determination model information received in operation S640. In an embodiment, the hub device 1000 may compare the calculated difference with the preset remaining memory capacity threshold, and may determine whether the calculated difference is greater than the preset threshold as a comparison result.

[0254] When it is determined that the difference between the remaining flash memory capacity and the capacity of the function determination model is greater than the preset remaining memory capacity threshold (operation S740), the hub device 1000 determines whether the simulated processing time of the function determination model is less than a preset processing time threshold. In an embodiment, the hub device 1000 may obtain information about the simulated processing time of the function determination model from the function determination model information received in operation S640. The hub device 1000 may compare the obtained simulated processing time of the function determination model with the preset processing time threshold, and may determine whether the simulated processing time of the function determination model is less than the preset threshold as a comparison result.

[0255] When it is determined that the difference between the remaining capacity of the flash memory and the capacity of the function determination model is less than a preset remaining memory capacity threshold (operation S670), the hub device 1000 receives access information of the function determination model corresponding to the new device from the voice assistant server 2000.

[0256] When it is determined that the simulation processing time of the function determination model is less than a preset processing time threshold (operation S660), the hub device 1000 can receive the function determination model corresponding to the new device from the voice assistant server 2000, and can store the function determination model in the internal memory.

[0257] When it is determined that the simulation processing time of the function determination model is greater than a preset processing time threshold (operation S670), the hub device 1000 receives access information of the function determination model corresponding to the new device from the voice assistant server 2000.

[0258] ​ It is a flowchart of a method for controlling the operation of a new device based on a user's voice input executed by the hub device 1000 according to an embodiment. ​ Including a series of operations executed after operation S660 in ​ and ​ The operations in. ​ Operation S810 in is an operation after the hub device 1000 receives the function determination model corresponding to the new device from the voice assistant server 2000 (see ​ ) and stores the received function determination model corresponding to the new device in the internal memory.

[0259] In operation S810, the hub device 1000 requests the IoT server 3000 (see ​ ) for a control command conversion module to convert the operation information about the new device into a control command. The operation information about the new device is information obtained by interpreting the user's voice input using the function determination model corresponding to the new device, and can be information related to the detailed operations to be performed by the new device, the relationship between each detailed operation and another detailed operation, and the execution order of the detailed operations. The control command conversion module can be a module for converting the operation information into instructions or an instruction set readable or executable by the new device, so that the new device executes the detailed operations included in the operation information obtained by the hub device 1000. In an embodiment, the hub device 1000 can send a query for the identification information (e.g., the id of the new device) of the new device and the control command conversion module to the IoT server 3000.

[0260] In operation S820, the hub device 1000 receives a control command conversion module for a new device from the IoT server 3000. In an embodiment, the hub device 1000 may store the received control command conversion module in the memory 1300 (see ​ ).

[0261] In operation S830, the hub device 1000 receives a voice input from a user. In an embodiment, the hub device 1000 may receive a voice input (e.g., the user's words) from the user through the microphone 1100 (see ​ ), and may obtain a voice signal from the received voice input. In an embodiment, the processor 1200 of the hub device 1000 (see ​ ) may convert the sound received through the microphone 1100 into an acoustic signal, and may obtain a voice signal by removing noise (e.g., non-speech components) from the acoustic signal.

[0262] In operation S840, the hub device 1000 converts the received voice input into text by performing automatic speech recognition (ASR). In an embodiment, the processor 1200 of the hub device 1000 may perform ASR to convert the voice signal into computer-readable text by using a predefined model such as an acoustic model (AM) or a language model (LM). When the hub device 1000 receives an acoustic signal without noise removal, the processor 1200 may obtain a voice signal by removing noise from the received acoustic signal, and may perform ASR on the voice signal.

[0263] In operation S850, the hub device analyzes the text by using an updated device determination model, and determines a new device as an operation execution device related to the analyzed text. In an embodiment, the processor 1200 of the hub device 1000 may analyze the text by using an updated first NLU model included in the updated device determination model, and may determine a new device as an operation execution device for performing an operation according to the user intention from a plurality of devices. The plurality of devices refers to devices that are logged in with the same user account as the hub device 1000 and are connected to the hub device 1000 through a network. The plurality of devices may be devices registered in the IoT server with the same user account as the hub device 1000.

[0264] The first NLU model is a model trained to analyze text converted from voice input and determine an operation execution device based on the analysis result. The first NLU model can be used to determine an intention by interpreting the text and determine an operation execution device based on the intention. The hub device 1000 can parse the text in units of morphemes, words, or phrases by using the first NLU model, and can infer the meaning of words extracted from the parsed text by using the linguistic features (e.g., grammatical components) of the morphemes, words, or phrases. The processor 1200 can compare the inferred meaning of the words with predefined intentions provided by the first NLU model and can determine the intention corresponding to the inferred meaning of the words.

[0265] The hub device 1000 can determine, based on a matching model for determining the relationship between an intention and a device, a device related to the intention identified from the text as an operation execution device. In an embodiment, the matching model can be obtained through learning by a rule-based system, but the present disclosure is not limited thereto.

[0266] In an embodiment, the hub device 1000 can obtain a plurality of numerical values indicating the degree of relationship between an intention and a plurality of devices by applying the matching model to the intention, and can determine, as the final operation execution device, the device having the maximum value among the obtained plurality of numerical values. For example, when an intention is related to each of a first device and a second device, the hub device 1000 can obtain a first numerical value indicating the degree of relationship between the intention and the first device and a second numerical value indicating the degree of relationship between the intention and the second device, and can determine the first device having the larger value among the first numerical value and the second numerical value as the operation execution device.

[0267] Although the hub device 1000 can train a matching model between an intention and an operation execution device by using, for example, a rule-based system, the present disclosure is not limited thereto. The AI model used by the hub device 1000 can be, for example, a neural network-based system (e.g., a convolutional neural network (CNN) or a recurrent neural network (RNN)), a support vector machine (SVM), linear regression, logistic regression, naive Bayes, random forest, decision tree, or k-nearest neighbor algorithm. Alternatively, the AI model can be a combination of the above examples or any other AI model.

[0268] The updated device determination model can be a model trained to determine an operation execution device from among a plurality of pre-registered devices and newly registered devices according to the user account of the hub device 1000. In an embodiment, the updated device determination model can be a model updated through learning or the like to analyze text about a new device, obtain an intention as an analysis result of the text, and determine the new device as an operation execution device based on the obtained intention.

[0269] The updated device determination model can analyze and calculate a numerical value indicating the degree of relationship between the intention obtained from the text and each of the newly registered new device and multiple pre-registered devices logged in using the same user account as the hub device 1000, and can determine the device with the maximum numerical value as the operation execution device.

[0270] In an embodiment, the hub device 1000 can receive device information about each of the newly registered new device and multiple pre-registered devices according to the user account from the voice assistant server 2000 (see ​ ). The device information may include at least one of the following: for example, identification information (e.g., device ID information) of the new device and each of the multiple devices, device type, function execution ability, location information, or status information. The hub device 1000 can determine, based on the received device information, a device for performing an operation from the new device and the multiple devices according to the intention. In an embodiment, the hub device 1000 can determine the new device as the operation execution device by analyzing the text using the updated device determination model.

[0271] In operation S860, the hub device 1000 provides at least part of the text to the function determination model corresponding to the new device stored in the internal memory. Although ​ not shown, in operation S860, the hub device 1000 can select the function determination model corresponding to the new device from one or more function determination models pre-stored in the internal memory. The "function determination model corresponding to the new device" refers to a model for obtaining operation information about the detailed operations for performing an operation according to the function determined for the new device determined as the operation execution device, the relationships between the detailed operations, and the execution order of the detailed operations.

[0272] The hub device 1000 provides at least part of the text to the selected function determination model. In an embodiment, the processor 1200 of the hub device 1000 (see ​ ) can provide at least part of the text rather than all of the text to the selected function determination model. For example, when the new device is an air purifier and the text converted from voice input is "Execute the deodorization mode in the air purifier", the phrase "in the air purifier" specifies the name of the operation execution device, so for the function determination model corresponding to the new device, the phrase "in the air purifier" may be unnecessary information. The processor 1200 can parse the text in units of words or phrases using the updated first NLU model, can identify words or phrases specifying names, general names, or device installation locations, and can provide the part of the text other than the identified words or phrases to the function determination model.

[0273] In operation S870, the hub device 1000 interprets at least part of the text by using a function determination model corresponding to the new device, and obtains operation information regarding operations to be performed by the new device. In an embodiment, the processor 1200 (see ​ ) of the hub device 1000 may analyze at least part of the text by using a second NLU model included in the function determination model corresponding to the new device. The processor 1200 may parse the text in units of morphemes, words, or phrases by using the second NLU model, may identify the meanings of the morphemes, words, or phrases parsed through syntactic or semantic analysis, and may determine an intention and parameters by matching the identified meanings with predefined words. As used herein, "parameters" refer to variable information for determining detailed operations of an operation execution device related to the intention. When the text provided to the function determination model corresponding to the new device is "Execute the deodorization mode", the intention may be "Operation mode execution", and the parameter may be the "deodorization mode" as the mode to be executed.

[0274] The processor 1200 (see ​ ) may obtain operation information regarding at least one detailed operation related to the intention and parameters by using an action planning management model of the function determination model corresponding to the new device. The action planning management model may manage information regarding detailed operations of the new device determined to be an operation execution device and the relationships between the detailed operations. The processor 1200 may plan detailed operations to be performed by the new device and the execution order of the detailed operations based on the intention and parameters by using the action planning management model, and thus may obtain operation information.

[0275] The operation information may be information related to detailed operations to be performed by the new device and the execution order of the detailed operations. The operation information may include information related to detailed operations to be performed by the new device, the relationships between each detailed operation and another detailed operation, and the execution order of the detailed operations. The operation information may include, but is not limited to, functions executed by the new device to perform a specific operation, the execution order of the functions, input values required to execute the functions, and output values output as execution results of the functions.

[0276] In operation S880, the hub device 1000 obtains a control command based on the obtained operation information. A control command refers to an instruction or set of instructions that can be read or executed by the new device so that the new device performs detailed operations included in the operation information.

[0277] ​ is a flowchart of an operation method of the hub device 1000, the voice assistant server 2000, the IoT server 3000, and the new device 4300 according to an embodiment.

[0278] ​Shows the operations of entities in a multi-device system environment including a hub device 1000, a voice assistant server 2000, an IoT server 3000, and an operation execution device 4000a. Refer to ​ , the hub device 1000 may include an ASR module 1310, an NLG module 1320, a TTS module 1330, a device determination model 1340, a resource tracking module 1350, and a function determination model management module 1360. In ​ , the hub device 1000 may not store a function determination model, or may not store a function determination model corresponding to a new device.

[0279] ​ The ASR module 1310, NLG module 1320, TTS module 1330, device determination model 1340, resource tracking module 1350, and function determination model management module 1360 in ​ are the same as these modules shown in

[0280] The voice assistant server 2000 may store a device determination model 2330, a voice assistant model update module 2340, a device-side model update module 2350, and multiple function determination models 2362, 2364, and 2368. For example, the function determination model 2362, which is the first function determination model stored in the voice assistant server 2000, may be a model for determining the functions of an air conditioner and obtaining operation information regarding the detailed operations related to the determined functions and the relationships between the detailed operations. For example, the function determination model 2364, which is the second function determination model, may be a model for determining the functions of a TV and obtaining operation information regarding the detailed operations related to the determined functions and the relationships between the detailed operations, and the function determination model 2368 may be a model for determining the functions of an air purifier as a new device and obtaining operation information regarding the detailed operations related to the determined functions and the relationships between the detailed operations.

[0281] In ​ , it may be determined that the operation execution device 4000a is an "air purifier" as a new device 4300, and the function determination model 2368 corresponding to the air purifier may be stored in the voice assistant server 2000.

[0282] In operation S910, the new device 4300 sends the device information and user account information of the new device 4300 to the IoT server 3000. When a user of the hub device 1000 obtains the new device 4300, turns on the new device 4300, and then logs in to the new device 4300 by entering the user account, the new device 4300 can send the user account information and device information obtained during the login process to the IoT server 3000. The device information sent to the IoT server 3000 may include at least one of the following: for example, the identification information of the new device (e.g., the id information of the new device 4300), the device type of the new device 4300, the function execution ability information of the new device 4300, the location information, or the status information.

[0283] In operation S920, the IoT server 3000 registers the new device 4300 by using the received user account. The IoT server 3000 can classify and register multiple devices according to the user account.

[0284] The new device 4300 (or the first device) may refer to a device newly obtained by the same user as the user of the hub device 1000 and registered in the IoT server 3000 by logging in with the same user account as the hub device 1000.

[0285] In operation S922, the IoT server 3000 sends the user account information and the device information of the new device 4300 registered by using the user account to the voice assistant server 2000.

[0286] In operation S930, the IoT server 3000 sends the device information of the new device 4300 to the hub device 1000. The IoT server 3000 can identify the hub device 1000 pre-registered by using the received user account, and can send the device information of the new device 4300 to the identified hub device 1000.

[0287] In operation S932, the voice assistant server 2000 sends the device information of the new device 4300 to the hub device 1000. The voice assistant server 2000 can identify the hub device 1000 pre-registered by using the user account received from the IoT server 3000, and can send the device information of the new device 4300 to the identified hub device 1000.

[0288] In operation S940, the hub device 1000 determines whether the device determination model 1340 needs to be updated. In an embodiment, the hub device 1000 can determine the storage in the memory 1300 (see ​) The device in determines whether the device determination model 1340 can determine the new device 4300 as the operation execution device according to the user's voice input. For example, the hub device 1000 can analyze the text converted from the user's voice input by using the first NLU model 1342 included in the device determination model 1340, and can determine whether there is discourse data for obtaining an intention related to the function of the new device 4300.

[0289] When it is determined that the device determination model 1340 related to the new device 4300 does not need to be updated (operation S944), the hub device 1000 determines the operation execution device by using the device determination model 1340. In operation S944, since the device determination model 1340 is pre-stored in the memory 1300 (see ​ ) before the hub device 1000 receives the device information of the new device 4300, the device determination model 1340 can obtain the intention related to the new device 4300 from the text even without a separate update process, and can determine the new device 4300 as the operation execution device based on the obtained intention.

[0290] When it is determined that the device determination model 1340 related to the new device 4300 needs to be updated (operation S942), the hub device 1000 sends the user account information, the identification information of the hub device 1000, and a query for requesting an update of the device determination model to the voice assistant server 2000.

[0291] In operation S950, the voice assistant server 2000 updates the device determination model 2330 by using the device information of the new device 4300. The voice assistant server 2000 can update the device determination model 2330 to a new model by using, for example, the voice assistant model update module 2340 through learning, so that the device determination model 2330 interprets the intention from the text based on the received device information of the new device 4300, and determines the new device 4300 as the operation execution device related to the intention as a result of the interpretation. In an embodiment, the voice assistant model update module 2340 can update the first NLU model to a new model through learning, so that the first NLU model of the device determination model 2330 interprets the text related to the new device 4300.

[0292] In operation S952, the voice assistant server 2000 provides the updated device determination model 2330 to the hub device 1000. In an embodiment, the voice assistant server 2000 can send at least one file constituting the updated device determination model 2330 to the hub device 1000.

[0293] In operation S960, the hub device 1000 stores the updated device determination model 2330 in the memory 1300 (see​ ) In an embodiment, the updated device determination model 2330 received from the voice assistant server 2000 can replace the device determination model 1340 pre-stored in the hub device 1000 by using an overwrite.

[0294] In operation S970, the voice assistant server 2000 obtains function determination model information corresponding to the new device 4300 by using the device information of the new device 4300. In an embodiment, the voice assistant server 2000 can obtain the function determination model 2368 corresponding to the new device 4300 by using the device information of the new device 4300 received from the IoT server 3000 in operation S922, and can store the obtained function determination model 2368 in the memory 2300 in addition to the function determination models 2362 and 2364 (see ​ ).

[0295] In operation S972, the voice assistant server 2000 provides the information of the function determination model corresponding to the new device 4300 to the hub device 1000. The information of the function determination model may include at least one of the following: for example, the capacity information of the function determination model 2368, the RAM occupancy information for reading the function determination model 2368, or the information of the simulation processing time required for the processor 1200 of the hub device 1000 (see ​ ) to obtain operation information according to the detailed operations and detailed operation sequences of the functions by reading the function determination model 2368.

[0296] In operation S980, the hub device 1000 identifies the capacity of the function determination model 2368 from the information of the function determination model corresponding to the new device 4300. In an embodiment, the hub device 1000 can identify the capacity of at least one file constituting the function determination model 2368 from the function determination model information received from the voice assistant server 2000. However, the present disclosure is not limited thereto, and the hub device 1000 can identify the RAM occupancy information for reading the function determination model 2368 from the information of the function determination model and the information of the simulation processing time required for the processor 1200 of the hub device 1000 (see ​ ) to read the function determination model 2368.

[0297] In operation S990, the hub device 1000 determines whether to store the function determination model 2368 corresponding to the new device 4300 in the memory 1300 of the hub device 1000 based on the received information of the function determination model 2368 corresponding to the new device 4300 (see ​ ). Operation S990 is related to ​The operations in [operation S650] are the same (including operations S710 to S740), and thus repeated descriptions will be omitted.

[0298] ​ is a flowchart of an operation method of the hub device 1000 and the new device 4300 according to an embodiment.

[0299] ​ shows entities in a multi-device system environment including the hub device 1000 and the new device 4300 in ​ the steps after the operation of the flowchart. ​ the steps in indicates a state in which the hub device 1000 determines to store the function determination model 2368 corresponding to the new device 4300 in the hub device 1000.

[0300] In operation S1010, the hub device 1000 may download the function determination model 2368 corresponding to the new device 4300 from the voice assistant server 2000, and may store the function determination model 2368 in the memory 1300 (see ​ ). Referring to ​ , the hub device 1000 may include a function determination model 1378 of the new device. In operation S1010, the hub device 1000 may send a query for requesting to send the function determination model 2368 corresponding to the new device 4300 to the voice assistant server 2000, may download the function determination model 2368 from the voice assistant server 2000, and may store the function determination model 2368 in the memory 1300. The function determination model 1378 of the new device included in the hub device 1000 may be the same as the function determination model 2368 corresponding to the new device 4300 received from the voice assistant server 2000.

[0301] In operation S1020, the hub device 1000 receives a voice input from a user.

[0302] In operation S1030, the hub device 1000 converts the received voice input into text by performing ASR.

[0303] In operation S1040, the hub device 1000 analyzes the text by using the updated device determination model 1340, and determines the new device 4300 as an operation execution device related to the analyzed text.

[0304] In operation S1050, the hub device 1000 provides at least part of the text to the function determination model 1378 of the new device stored in the memory 1300.

[0305] In operation S1060, the hub device 1000 obtains operation information regarding operations to be performed by the new device 4300 by interpreting at least part of the text using the function determination model 1378 of the new device.

[0306] Operations S1020 to S1060 are the same as operations S830 to S870 in ​ and thus repeated descriptions will be omitted.

[0307] In operation S1062, the hub device 1000 sends the operation information by using the identification information of the new device 4300. In an embodiment, the hub device 1000 can identify the new device 4300 by using the identification information of the new device 4300 (e.g., the id information of the new device 4300), and can send the operation information to the identified new device 4300.

[0308] In operation S1070, the new device 4300 obtains a control command based on the obtained operation information. In operation S1070, the new device 4300 itself can include a control command conversion module for converting the operation information into a control command. The new device 4300 can convert the operation information into an instruction or set of instructions that can be read or executed by the new device 4300 by using the control command conversion module.

[0309] In operation S1080, the new device 4300 performs operations according to the obtained control command.

[0310] In an embodiment, after performing the operation, the new device 4300 can send information regarding the operation execution result to the hub device 1000. However, the present disclosure is not limited thereto, and the new device 4300 can send information regarding the operation execution result to the IoT server 3000 (see ​ ).

[0311] ​ is a flowchart of an operation method of the hub device 1000, the voice assistant server 2000, the IoT server 3000, and the new device 4300 according to an embodiment.

[0312] ​ shows the entities in a multi-device system environment including the hub device 1000, the voice assistant server 2000, the IoT server 3000, and the new device 4300 in ​ the steps after ​ the steps in indicate a state where the hub device 1000 does not store the function determination model 2368 corresponding to the new device 4300 in the hub device 1000. Since in step it is determined not to store the function determination model 2368 corresponding to the new device 4300 in the memory 1300 (see ​ ), so different from the hub device 1000 in ​ , the hub device 1000 in ​ does not include the function determination model 1378 of the new device.

[0313] In operation S1110, the hub device 1000 sends a query for requesting access information of the function determination model 2368 corresponding to the new device 4300 to the voice assistant server 2000. The access information may include at least one of the following: for example, identification information (e.g., server id) of the voice assistant server 2000 storing the function determination model 2368 corresponding to the new device 4300, location information, IP address information, MAC address, application programming interface (API) information accessible to the function determination model 2368 in the voice assistant server 2000, the language used by the function determination model 2368, or identification information of the corresponding new device 4300. In operation S1110, the hub device 1000 may send the identification information and user account information of the hub device 1000 to the voice assistant server 2000.

[0314] In operation S1120, the voice assistant server 2000 obtains the access information of the function determination model 2368 corresponding to the new device 4300.

[0315] In operation S1122, the voice assistant server 2000 sends the access information of the function determination model 2368 corresponding to the new device 4300 to the hub device 1000.

[0316] In operation S1130, the hub device 1000 stores the received access information in the memory 1300 (see ​ ).

[0317] In operation S1140, the hub device 1000 receives the user's voice input.

[0318] In operation S1150, the hub device 1000 converts the received voice input into text by performing ASR.

[0319] In operation S1160, the hub device 1000 analyzes the text by using the updated device determination model 1340, and determines the new device 4300 as the operation execution device corresponding to the analyzed text.

[0320] Operations S1140 to S1160 are the same as operations S830 to S850 in ​ , so the repeated description will be omitted.

[0321] In operation S1162, the hub device 1000 sends at least part of the text to the function determination model 2368 corresponding to the new device 4300 based on the access information. In an embodiment, the hub device 1000 may identify the voice assistant server 2000 storing the function determination model 2368, and the voice assistant server 2000 stores at least one of the identification information (e.g., server id), location information, IP address information, MAC address, API information accessible to the function determination model 2368 in the voice assistant server 2000, the language used by the function determination model 2368, or the identification information of the new device 4300, and can send at least part of the text to the function determination model 2368 of the voice assistant server 2000.

[0322] In operation S1170, the voice assistant server 2000 interprets at least part of the text by using the function determination model 2368 corresponding to the new device 4300, and obtains operation information about the operation to be performed by the new device 4300. In an embodiment, the voice assistant server 2000 selects the function determination model 2368 corresponding to the new device 4300 from the multiple function determination models 2362, 2364, and 2368, interprets the text by using the NLU model 2368a of the selected function determination model 2368, and determines the intent based on the analysis result. The voice assistant server 2000 may analyze at least part of the text received from the hub device 1000 by using the NLU model 2368a of the function determination model 2368. The NLU model 2368a, as an AI model trained to interpret text related to the new device 4300, may be a model trained to determine the intent and parameters related to the operation desired by the user. The NLU model 2368a may be a model trained to determine the functions related to the type of the new device 4300 when the input text is received.

[0323] In an embodiment, the voice assistant server 2000 may parse at least part of the text in units of words or phrases by using the NLU model 2368a, may infer the meaning of the words extracted from the parsed text by using the language features (e.g., grammatical elements) of the parsed morphemes, words, or phrases, and may obtain the intent and parameters from the text by matching the inferred meaning with the predefined intent and parameters. In an embodiment, the voice assistant server 2000 may determine only the intent from at least part of the text.

[0324] The voice assistant server 2000 obtains operation information regarding operations to be performed by the new device 4300 based on the intent. In an embodiment, the voice assistant server 2000 plans the operation information to be performed by the new device 4300 based on the intent and parameters by using the action planning management model 2368b of the function determination model 2368. The action planning management model 2368b can interpret the operations to be performed by the new device 4300 based on the intent and parameters. The action planning management model 2368b can select detailed operations related to the interpreted operations from the operations of the pre-stored devices, and can plan the execution order of the selected detailed operations. The action planning management model 2368b can obtain operation information regarding the detailed operations to be performed by the new device 4300 by using the planning result.

[0325] In operation S1172, the voice assistant server 2000 sends the obtained operation information and identification information of the new device 4300 to the IoT server 3000.

[0326] In operation S1180, the IoT server 3000 obtains a control command based on the identification information of the new device 4300 and the received operation information. The IoT server 3000 can include a database storing control commands and operation information of multiple devices. In an embodiment, the IoT server 3000 can select, based on the identification information of the new device 4300, a control command for controlling the detailed operations of the new device 4300 from the control commands related to the multiple devices pre-stored in the database.

[0327] In operation S1182, the IoT server 3000 sends the control command to the new device 4300 by using the identification information of the new device 4300.

[0328] In operation S1190, the new device 4300 performs an operation corresponding to the received control command.

[0329] In an embodiment, after performing the operation, the new device 4300 can send information regarding the operation execution result to the IoT server 3000.

[0330] ​ is a conceptual diagram showing operations of a hub device and a listening device according to an embodiment;

[0331] ​ The arrows in indicate the movement, transmission, and reception of data including voice signals and text between the first device 4100a and the second device 4200a. The circled numbers indicate the operation order.

[0332] The first device 4100a and the second device 4200a can be connected to each other and communicate with each other by using a wired or wireless communication method. In an embodiment, the first device 4100a and the second device 4200a can be directly connected to each other through a communication network, but the present disclosure is not limited thereto. In an embodiment, the first device 4100a and the second device 4200a can be connected to the voice assistant server 2000 (see ​ ), and can be connected to each other through the voice assistant server 2000.

[0333] Reference ​ , the first device 4100a can be a listening device that receives voice input from a user, and the second device 4200b can be a hub device that determines an operation execution device by interpreting the voice input and controls the determined operation execution device to execute an operation. For example, the first device 4100a can be an air conditioner, and the second device 4200a can be a TV.

[0334] The first device 4100a can be a listening device that receives voice input including speech from a user. The listening device can be a device that only receives voice input from a user, but the present disclosure is not limited thereto. In an embodiment, the listening device can be an operation execution device that executes an operation related to a specific function by receiving a control command from a hub device ( ​ the second device 4200a in).

[0335] In an embodiment, the listening device can receive voice input related to the function executed by the listening device from a user. For example, the first device 4100a receives voice input such as "lower the air conditioner temperature to 20 °C" from a user (step ①).

[0336] In an embodiment, the first device 4100a can convert the sound received through a microphone into an acoustic signal, and can obtain a voice signal by removing noise (e.g., non-speech components) from the acoustic signal.

[0337] The first device 4100a sends the voice signal to the second device 4200a as a hub device (step ②).

[0338] The second device 4200a as a hub device receives the voice signal from the first device 4100a, converts the voice signal into text, and determines the listening device as an operation execution device by interpreting the text (step ③). In an embodiment, the second device 4200a can perform ASR by using data of an automatic speech recognition (ASR) module pre-stored in a memory, and convert the voice signal into text. The second device 4200a can detect an intention from the text by using a device determination model pre-stored in the memory, and can determine a device for executing an operation corresponding to the detected intention. In​ In this case, the second device 4200a may determine the first device 4100a as the operation execution device.

[0339] The function determination model corresponding to the operation execution device determined by the second device 4200a, which serves as a hub device, may be pre-stored in the memory of the hub device, may be stored in the first device 4100a itself determined as the operation execution device, or may be stored in the memory 2300 of the voice assistant server 2000 (see ​ ) (see ​ ). The "function determination model" corresponding to each device is a model for obtaining operation information regarding detailed operations for performing operations according to the determined functions of the devices and the relationships between the detailed operations.

[0340] In ​ , the function determination model corresponding to the first device 4100a may be pre-stored in the memory of the first device 4100a. For example, the first device 4100a may store the function determination model 4132 (see ​ ), where the function determination model 4132 is used to obtain operation information regarding detailed operations corresponding to the air conditioner and the relationships between the detailed operations.

[0341] The second device 4200a, which serves as a hub device, provides the text to the function determination model of the listening device (step ④).

[0342] The first device 4100a performs operations by interpreting the received text (step ⑤). In an embodiment, the first device 4100a may analyze at least part of the text by using the NLU model 4134 included in the function determination model 4132 (see ​ ), and may obtain operation information regarding the operations to be performed by the first device 4100a based on the analysis result of at least part of the text. The function determination model 4132 may include an action planning management model 4136 (see ​ ), where the action planning management model 4136 is configured to manage operation information related to the detailed operations of the device to generate the detailed operations to be performed by the first device 4100a and the execution order of the detailed operations. The action planning management model 4136 may manage operation information regarding the detailed operations performed by the first device 4100a and the relationships between the detailed operations. The action planning management model 4136 may plan the detailed operations to be performed by the first device 4100a and the execution order of the detailed operations based on the analysis result of at least part of the text. The first device 4100a may plan the detailed operations and the execution order of the detailed operations based on the analysis result of the function determination model 4132 for at least part of the text, and may perform operations based on the planning result.

[0343] ​ is a conceptual diagram showing the operations of a hub device and a second device according to an embodiment.

[0344] ​ The arrows in indicate the movement, transmission, and reception of data including voice signals and text between the first device 4100b and the second device 4200b. The circled numbers indicate the order of operations.

[0345] Similar to ​ the ​ first device 4100b and the second device 4200b in can be connected to each other by using a wired or wireless communication method and can perform communication.

[0346] Referring to ​ , the first device 4100b can be a listening device that receives voice input from a user, and can be a hub device that determines an operation execution device by interpreting the received voice input and controls the determined operation execution device to perform an operation.

[0347] In ​ , the first device 4100b receives a voice input such as "lower the air conditioner temperature to 20°C" from the user (step ①).

[0348] The first device 4100b determines the hub device as the operation execution device by interpreting the voice input and performs an operation (step ②). The method by which the first device 4100b, as a hub device, converts the voice input into text, determines the operation execution device by interpreting the converted text, obtains operation information for performing the operation of the operation execution device, and performs the operation based on the obtained operation information is the same as that of the first device 4100a in ​ , so repeated descriptions will be omitted.

[0349] In ​ and ​ , the operation execution devices related to the discourse intention of an operation performed by the user through a specific device (for example, the operation intention of lowering the set temperature of the air conditioner) are both the first devices 4100a and 4100b. However, since the first device 4100a is a listening device in ​ , the first device 4100a may not be able to determine the operation execution device by directly interpreting the voice input, so a process of sending the voice signal to the second device 4200a, which is a hub device, and receiving text from the second device 4200a is also required. However, since the first device 4100b is a hub device in ​ , unnecessary processes of sending the voice signal to the second device 4200b or receiving text from the second device 4200b are omitted. Therefore, in ​ , the waiting time can be less than ​the waiting time therein, and the speech processing time may be less than ​ the speech processing time therein. The "speech processing time" refers to the time from receiving the user's speech to completing the operation execution. In an embodiment, the speech processing time may refer to the time from receiving the user's speech to outputting a message indicating that the operation execution is completed.

[0350] In an embodiment, a device that is more frequently used by users among multiple devices and requires a short speech processing time may be determined as the hub device. Additionally, when there is a first device that is more frequently used by the user and requires a short speech processing time than a second device currently used as the hub device, the hub device may be changed by replacing it with the first device. In ​ and ​ , since the frequency of using the air conditioner increases in summer, the frequency of speech related to air conditioner control increases. Considering the usage frequency and speech processing time, the second device 4200a, which is a TV, may no longer be used as the hub device, and instead, the first device 4100b, which is an air conditioner (see ​ ), may be used as the hub device to reduce the waiting time. Additionally, in an abnormal situation where the hub device may not operate properly (such as the device currently used as the hub device shutting down or losing the network connection), the hub device may be replaced by another device to flexibly handle the abnormal situation.

[0351] In an embodiment, the hub device may be periodically changed by periodically obtaining information on the speech processing time and usage frequency of each of the multiple devices at a preset time interval and comparing the speech processing time and usage frequency of the device currently used as the hub device with the obtained information.

[0352] ​ is a block diagram of a hub device 1002 according to an embodiment.

[0353] The hub device 1002 is a device that receives the user's voice input and controls at least one of the multiple devices 4000 based on the received voice input. The hub device 1002 may be a listening device that receives the user's voice input.

[0354] Referring to ​ , the hub device 1002 may at least include a microphone 1100, a processor 1200, a memory 1300, and a communication interface 1400. The microphone 1100, the processor 1200, and the communication interface 1400 among the elements of the hub device 1002 are the same as those shown in ​ , and thus repeated descriptions will be omitted.

[0355] The memory 1300 may store data about the ASR module 1310, data about the NLG module 1320, data about the TTS module 1330, data about the device determination model 1340, data about the resource tracking module 1350, data about the function determination model management module 1360, data about multiple function determination models 1370, and data about the hub device determination module 1380. The ASR module 1310, the NLG module 1320, the TTS module 1330, the device determination model 1340, the resource tracking module 1350, the function determination model management module 1360, and the multiple function determination models 1370 are connected to the hub device determination module 1380. ​ The same as shown, so repeated description will be omitted.

[0356] The hub device determination module 1380 is configured to determine the hub device 1002 and the user account that is pre-registered in the IoT server 3000 (see ​ ) in a plurality of devices 4000 (see ​ ) as a hub device. The hub device determination module 1380 may include data, instructions, or program codes configured to: select at least one candidate hub device from the plurality of devices 4000; determine a device from the at least one candidate hub device based on usage history information and performance information of each of the at least one candidate hub device; and change the hub device by replacing the current hub device 1002 with the determined device.

[0357] The processor 1200 may select at least one candidate hub device from the plurality of devices 4000 by using data or program code related to the hub device determination module 1380. In an embodiment, the processor 1200 may obtain at least one item of information on power supply constancy, computing capability, or power consumption of each of the plurality of devices 4000 by using data or program code related to the hub device determination module 1380, and may select at least one candidate hub device from the plurality of devices 4000 based on at least one item of power supply constancy, computing capability, or power consumption. "Power supply constancy" refers to whether the power supplied to each of the plurality of devices 4000 is constant and continuous.

[0358] The processor 1200 may obtain usage history information and performance information of each of at least one candidate hub device by using data or program code regarding the hub device determination module 1380, and may determine one device from the at least one candidate hub device based on the obtained usage history information and performance information. The processor 1200 may change the hub device by replacing the current hub device 1002 with the determined device. For example, when the current hub device 1002 is a TV, the processor 1200 may determine an air conditioner among the plurality of devices 4000 as a new hub device by using data regarding the hub device determination module 1380, and may change the hub device 1002 by replacing the TV, which is the current hub device 1002, with the air conditioner.

[0359] The hub device determination module 1380 may include a usage history log database 1382 and a device performance history database 1384. Although the usage history log database 1382 and the device performance history database 1384 are included in the hub device determination module 1380 in ​ this disclosure is not limited thereto. In an embodiment, the usage history log database 1382 and the device performance history database 1384 may not be included in the hub device determination module 1380 and may be stored in the memory 1300.

[0360] In another embodiment, at least one of the usage history log database 1382 or the device performance history database 1384 may not be included in the hub device 1002 and may include an external database. In this case, the hub device 1002 may access at least one of the usage history log database 1382 or the device performance history database 1384 through a network.

[0361] The usage history log database 1382 is a database that stores information on the usage frequency and the most recent usage history of the hub device 1002 and each of the plurality of devices 4000 in the form of a log. The usage history log database 1382 may store at least one of the following data in the form of a log: the frequency at which the hub device 1002 and each of the plurality of devices 4000 are used as a listening device by the user, the frequency at which the hub device 1002 and each of the plurality of devices 4000 are used as an operation execution device, the most recent usage history of the hub device 1002 and each of the plurality of devices 4000 being used as a listening device, or the most recent usage history of the hub device 1002 and each of the plurality of devices 4000 being used as an operation execution device. The usage history log database 1382 may store, for example, voice inputs frequently received from the user or texts converted from the voice inputs.

[0362] The device performance history database 1384 is a database that stores the performance history information of the hub device 1002 and each of the multiple devices 4000 in the form of logs. The device performance history database 1384 can store data such as the speech processing time of the hub device 1002 and each of the multiple devices 4000 in the form of logs. The "speech processing time" refers to the time from when the device receives the user's speech to when the operation execution is completed. In an embodiment, the speech processing time may refer to the time from when the user's speech is received to when the operation execution completion message is output. The speech processing time is inversely proportional to the performance of the device. For example, as the speech processing time decreases, the performance of the device improves.

[0363] In an embodiment, the hub device 1002 can obtain information on the simulation results of the speech processing time from each of the multiple devices 4000. In an embodiment, each of the multiple devices 4000 can determine the operation execution device by interpreting the text of the user's speech such as "lower the air conditioner temperature to 20°C" or "play the movie Avengers on the TV", can obtain the operation information on the operation to be performed by the determined operation execution device, can generate a control command by using the obtained operation information, and can simulate the speech processing time required for the operation execution device to perform the operation based on the generated control command. In an embodiment, each of the multiple devices 4000 can simulate the speech processing time by using a background process. The "background process" refers to a process that the device itself executes in the background environment without user intervention. Each of the multiple devices 4000 can send the simulation results to the hub device 1002. The hub device 1002 can store the simulation results of the speech processing time obtained from each of the multiple devices 4000 in the device performance history database 1384.

[0364] The processor 1200 can analyze the usage history log database 1382 by using the data or program code of the hub device determination module 1380 to obtain the usage frequency information of the hub device 1002 and each of at least one candidate device. In addition, the processor 1200 can analyze the device performance history database 1384 by using the data or program code of the hub device determination module 1380 to obtain the speech processing time information of the hub device 1002 and each of at least one candidate hub device. The processor 1200 can determine the device for replacing the current hub device 1002 based on the usage frequency information of the hub device 1002 and each of at least one candidate hub device and the speech processing time information. The processor 1200 can change the hub device 1002 to a second device by using the data or program code of the hub device determination module 1380 to replace the first device serving as the current hub device 1002 with the newly determined second device. For reference ​Describe in detail the method by which the processor 1200 changes the hub device 1002 based on usage frequency information and speech processing time information.

[0365] However, the present disclosure is not limited to the processor 1200 that changes the hub device 1002 based on usage frequency information and speech processing time information. In an embodiment, the hub device 1002 may receive a user input for selecting one of a plurality of devices 4000, and may determine the selected device as the hub device based on the received user input. For example, the hub device 1002 may receive a voice input of a user for selecting one of a plurality of devices 4000 through the microphone 1100, and the processor 1200 may convert the voice input into text by using the data of the ASR module 1310, and may identify the device desired by the user by interpreting the text by using the data of the first NLU model 1342. The processor 1200 may change the hub device to the identified device.

[0366] ​ It is a flowchart of an operation method of the hub device 1002 according to an embodiment.

[0367] In operation S1410, the hub device 1002 selects at least one candidate hub device from a plurality of devices 4000 (see ​ ) pre-registered according to a user account. In an embodiment, the hub device 1002 may obtain at least one of the following information: the power supply constancy, computing power, or power consumption amount of each of a plurality of devices 4000 pre-registered in the IoT server 3000 (see ​ ) according to the user account logged in to the hub device 1002 and connected to the hub device 1002 through a network, and may determine at least one candidate hub device from the plurality of devices 4000 based on at least one of the power supply constancy, computing power, or power consumption amount.

[0368] In operation S1420, the hub device 1002 determines one device from the at least one candidate hub device based on the usage history information and performance information of each of the at least one candidate hub device.

[0369] In an embodiment, the hub device 1002 may analyze the usage history log database 1382 (see ​ ) stored in the memory 1300 (see ​) to obtain usage frequency information about each of the hub device 1002 and at least one candidate hub device. The hub device 1002 can obtain at least one of the following usage history information from the usage history log database 1382: for example, the frequency at which each of the hub device 1002 and at least one candidate hub device is used by the user as a listening device, the frequency at which each of the hub device 1002 and at least one candidate hub device is used as an operation execution device, the most recent usage history of each of the hub device 1002 and at least one candidate hub device being used as a listening device, and the most recent usage history of each of the hub device 1002 and at least one candidate hub device being used as an operation execution device.

[0370] In an embodiment, the hub device 1002 can obtain performance history information about each of the hub device 1002 and at least one candidate hub device by analyzing the device performance history database 1384 stored in the memory 1300 (see ​ ). The hub device 1002 can obtain, for example, speech processing time information about each of the hub device 1002 and at least one candidate hub device from the device performance history database 1384. "Speech processing time" refers to the time from receiving the user's speech to completing the operation execution. In an embodiment, the speech processing time can refer to the time from receiving the user's speech to outputting a message indicating that the operation execution is completed.

[0371] In an embodiment, the hub device 1002 can determine a device for replacing the hub device 1002 from at least one candidate hub device by comparing the usage frequency and speech processing time of at least one candidate hub device with the usage frequency and speech processing time of the current hub device 1002. For example, the hub device 1002 can determine a device from at least one candidate hub device by comparing the maximum value of the usage frequencies of at least one candidate hub device with the usage frequency of the current hub device 1002 and comparing the minimum value of the speech processing times of at least one candidate hub device with the speech processing time of the current hub device 1002. The detailed method of determining a device from at least one candidate hub device will be described in detail below with reference to ​ the operations S1510 to S1550 in.

[0372] In operation S1430, the hub device 1002 changes the hub device 1002 by replacing the current hub device 1002 with the determined device. For example, when the current hub device 1002 is a TV, the hub device 1002 can change the hub device 1002 by replacing the TV with an air conditioner determined from at least one candidate hub device.

[0373] ​ It is a flowchart of an operation method of the hub device 1002 according to an embodiment.

[0374] ​ The operations S1510 to S1550 in ​ are the detailed operations of the operation S1420 in ​ which are executed after the operation S1410 in ​ The operation S1510 in

[0375] In the operation S1510, the hub device 1002 obtains usage frequency information about each of the hub device 1002 and at least one candidate hub device by analyzing the usage history log database 1382 (see ​ ). In an embodiment, the processor 1200 of the hub device 1002 (see ​ ) can access the usage history log database 1382 stored in the memory 1300 (see ​ ), and can obtain usage frequency information including at least one of the following by analyzing the usage history log database 1382: the frequency at which each of the hub device 1002 and at least one candidate hub device is used by the user as a listening device, or the frequency at which each of the hub device 1002 and at least one candidate hub device is used as an operation execution device. Additionally, the processor 1200 can obtain information about the most recent usage history including at least one of the following by analyzing the usage history log database 1382: the most recent usage history of each of the hub device 1002 and at least one candidate hub device as a listening device, or the most recent usage history of each of the hub device 1002 and at least one candidate hub device as an operation execution device.

[0376] In the operation S1520, the hub device 1002 compares the usage frequency of the current hub device 1002 with the maximum value among the usage frequencies of at least one candidate hub device, and determines whether the usage frequency of the hub device 1002 is equal to or less than the maximum value among the usage frequencies of at least one candidate hub device. When there is only one candidate hub device, the hub device 1002 can determine whether the usage frequency of the device used as the current hub device 1002 is less than the usage frequency of the candidate hub device.

[0377] When it is determined that the usage frequency of the device used as the current hub device 1002 is less than the maximum value among the usage frequencies of at least one candidate hub device (operation S1530), the hub device 1002 analyzes the device performance history database 1384 (see ​)Obtain speech processing time information for each of the hub device 1002 and at least one candidate hub device. In an embodiment, the processor 1200 can access the device performance history database 1384 stored in the memory 1300, and can obtain the speech processing time information for each of the hub device 1002 and at least one candidate hub device by analyzing the device performance history database 1384. "Speech processing time" refers to the time from when the device receives the user's speech to when the operation execution is completed. In an embodiment, the speech processing time can refer to the time from when the user's speech is received to when the operation execution completion message is output. The speech processing time is inversely proportional to the performance of the device. For example, as the speech processing time decreases, the performance of the device improves.

[0378] In operation S1540, the hub device 1002 determines whether the speech processing time of the device serving as the current hub device 1002 is equal to or greater than the minimum value among the speech processing times of at least one candidate hub device by comparing the speech processing time of the device serving as the current hub device 1002 with the minimum value among the speech processing times of at least one candidate hub device. When there is only one candidate hub device, the hub device 1002 can determine whether the speech processing time of the device serving as the current hub device 1002 is greater than the speech processing time of the candidate hub device.

[0379] When it is determined that the speech processing time of the device serving as the current hub device 1002 is greater than the minimum value among the speech processing times of at least one candidate hub device (operation S1550), the hub device 1002 can determine the device for replacing the hub device 1002 based on the usage frequency information and speech processing time information about at least one candidate hub device. In an embodiment, the hub device 1002 can determine the device that has the maximum frequency and requires the shortest speech processing time when serving as at least one of the listening device or the operation execution device among at least one candidate hub device as the device for replacing the hub device 1002. In one embodiment, although the hub device 1002 considers both the usage frequency and the speech processing time, the hub device 1002 can assign a greater weight to the usage frequency to determine the device for replacing the hub device 1002. For example, the hub device 1002 can first select one or more devices with a relatively large usage frequency from at least one candidate hub device, and then can determine the device with the shortest speech processing time from the selected one or more devices.

[0380] However, the present disclosure is not limited thereto. The hub device 1002 can first select one or more devices whose required speech processing time is less than a preset threshold time from at least one candidate hub device, and then can determine the device with the maximum usage frequency from the selected one or more devices.

[0381] In operation S1430, the hub device 1002 changes the hub device by replacing the device serving as the current hub device 1002 with the determined device.

[0382] In operation S1520, when the usage frequency of the current hub device is greater than the maximum value among the usage frequencies of at least one candidate hub device (operation S1440), the device serving as the current hub device 1002 is maintained as the hub device. Similarly, in operation S1540, when the speech processing time of the current hub device 1002 is equal to or less than the minimum value among the speech processing times of at least one candidate hub device (operation S1440), the device serving as the current hub device 1002 is maintained as the hub device. When the usage frequency of the device serving as the current hub device 1002 is greater than that of another device and the speech processing time of the device serving as the current hub device 1002 is less than that of the device, the hub device does not need to be changed, so the device serving as the hub device 1002 is maintained.

[0383] ​ is a flowchart of an operation method of the hub device 1002 according to an embodiment.

[0384] ​ Operations S1610 to S1690 in ​ are operations performed between operations S610 and S620 in ​ Operation S1610 is the same as operation S610 in

[0385] In operation S1610, the hub device 1002 receives device information of a new device registered according to a user account from the IoT server 3000 (see ​ ). The new device refers to a device newly obtained by the user of the hub device 1002 through purchase or transfer of ownership, and registered in the IoT server 3000 by logging in with the same user account as the hub device 1002. The device information of the new device received by the hub device 1002 may include at least one of the following: for example, identification information of the new device (e.g., device id information), device type of the new device, function execution ability information of the new device, or location information. In an embodiment, the hub device 1000 may receive status information about the power on / off or current operation being performed of the new device.

[0386] The hub device 1002 may receive device information of a new device from the IoT server 3000. However, the present disclosure is not limited thereto, and the hub device 1002 may receive it from the voice assistant server 2000 (see​ )Receive device information of a new device.

[0387] In operation S1620, the hub device 1002 determines whether there is a hub device among the multiple devices pre-registered using a user account. In an embodiment, the voice assistant server 2000 (see ​ ) may determine the hub device 1002 based on the resource status information of the multiple devices pre-registered in the IoT server 3000 according to the same user account logged in to the hub device 1002. In an embodiment, the hub device may be a device including a device determination model 1340 among the multiple devices (see ​ ) and capable of determining an operation execution device by interpreting a voice input received from a user.

[0388] When it is determined that there is no hub device among the multiple devices (operation S1630), the new device is determined as the hub device 1002.

[0389] When it is determined that there is a hub device 1002 among the multiple devices (operation S1640), the hub device 1002 selects a candidate hub device based on a constant power supply, computing power, and power consumption amount among the multiple devices pre-registered using a user account. In an embodiment, the hub device 1002 may determine one or more candidate hub devices from the multiple devices. Operation S1640 is the same as ​ operation S1410 in, and thus repeated descriptions will be omitted.

[0390] In operation S1650, the hub device 1002 determines whether it receives a voice input for requesting a change of the hub device. For example, when the device serving as the current hub device 1002 is a TV, the hub device 1002 may receive a voice input for requesting a change of the hub device, such as "Change the air conditioner to the hub device". However, the present disclosure is not limited to receiving a voice input, and the hub device 1002 may receive an input from a user through a manipulation device such as a mouse or a keyboard or a touch input, and the manipulation device changes the hub by replacing the device serving as the current hub device 1002 with a new device.

[0391] When a voice input for requesting a change of the hub device is received (operation S1652), the hub device 1002 determines a device for replacing the hub device 1002 based on the received voice input. The hub device 1002 may convert the voice input into text, may interpret the text by using an NLU model to identify the name or type of the device desired by the user, and may determine the identified device as the device for replacing the hub device 1002. In an embodiment, the processor 1200 of the hub device 1002 (see ​)The received voice input can be converted into text by using data or program code related to the ASR module 1310 (see ​ ). The text can be parsed in units of words or phrases by using data or program code related to the first NLU model 1342 (see ​ ), words or phrases that identify the specified device name, common name, or device installation location can be recognized, and the device can be recognized by the recognized words or phrases. The device hub device 1002 can determine the recognized device as the device for replacing the hub device 1002.

[0392] When no voice input for requesting a change in the hub device is received from the user (operation S1660), the hub device 1002 obtains the speech processing time information about each of the hub device 1002 and at least one candidate hub device by analyzing the device performance history database 1384 (see ​ ).

[0393] In operation S1670, the hub device 1002 determines whether the speech processing time of the device used as the current hub device 1002 is greater than the minimum value of the speech processing times of at least one candidate hub device by comparing the speech processing time of the device used as the current hub device 1002 with the minimum value of the speech processing times of at least one candidate hub device.

[0394] Operations S1660 to S1670 are the same as operations S1530 and S1540 in ​ respectively, so the repeated description will be omitted.

[0395] When the speech processing time of the device used as the current hub device 1002 is equal to or less than the minimum value of the speech processing times of at least one candidate hub device (operation S1680), the device used as the current hub device 1002 is determined as the hub device 1002.

[0396] When it is determined that the speech processing time of the device used as the current hub device 1002 is greater than the minimum value of the speech processing times of at least one candidate hub device (operation S1682), the hub device 1002 determines the device for replacing the hub device 1002 based on the speech processing time information about at least one candidate hub device. In an embodiment, the hub device 1002 can determine the device with the shortest speech processing time among at least one candidate hub device as the device for replacing the hub device 1002.

[0397] In operation S620, the determined hub device 1002 requests the voice assistant server 2000 to update the device determination model 1340 stored in the hub device 1002 (see​ )。

[0398] ​ It is a diagram of a network environment including a hub device 1000, a plurality of devices 4000, and a voice assistant server 2000 according to an embodiment.

[0399] Reference ​ , the hub device 1000, the plurality of devices 4000, the voice assistant server 2000, and the IoT server 3000 can be connected and communicate with each other by using a wired or wireless communication method. In an embodiment, the hub device 1000 and the plurality of devices 4000 can be directly connected to each other through a communication network, but the present disclosure is not limited thereto.

[0400] The hub device 1000 and the plurality of devices 4000 can be connected to the voice assistant server 2000, and the hub device 1000 can be connected to the plurality of devices 4000 through a server. In addition, the hub device 1000 and the plurality of devices 4000 can be connected to the IoT server 3000. In another embodiment, each of the hub device 1000 and the plurality of devices 4000 can be connected to the voice assistant server 2000 through a communication network, and can be connected to the IoT server 3000 through the voice assistant server 2000. In another embodiment, the hub device 1000 can be connected to the plurality of devices 4000, and the hub device 1000 can be connected to the plurality of devices 4000 through one or more nearby access points. In addition, in a state where the hub device 1000 is connected to the voice assistant server 2000 or the IoT server 3000, the hub device 1000 can be connected to the plurality of devices 4000.

[0401] The hub device 1000, the plurality of devices 4000, the voice assistant server 2000, and the IoT server 3000 can be connected through a local area network (LAN), a wide area network (WAN), a value-added network (VAN), a mobile radio communication network, a satellite communication network, or a combination thereof. Examples of wireless communication methods may include, but are not limited to, Wi-Fi, Bluetooth, Bluetooth Low Energy (BLE), Zigbee, Wi-Fi Direct (WFD), Ultra-Wideband (UWB), Infrared Data Association (IrDA), and Near Field Communication (NFC).

[0402] In an embodiment, the hub device 1000 may receive a voice input from a user. At least one of the plurality of devices 4000 may be a target device that receives control commands from the voice assistant server 2000 and / or the IoT server 3000 and performs specific operations. At least one of the plurality of devices 4000 may be controlled to perform a specific operation based on the user voice input received by the hub device 1000. In an embodiment, at least one of the plurality of devices 4000 may receive a control command from the hub device 1000 and not from the voice assistant server 2000 and / or the IoT server 3000.

[0403] The hub device 1000 may receive a voice input (e.g., utterance) from a user. In an embodiment, the hub device 1000 may include an ASR model. In an embodiment, the hub device 1000 may include an ASR model with limited functions. For example, the hub device 1000 may include an ASR model having a function of detecting a specified voice input (e.g., a wake-up input such as "Hi, Bixby" or "OK, Google") or a function of preprocessing a voice signal obtained from a partial voice input. Although in ​ the hub device 1000 is an AI speaker in the example, the present disclosure is not limited thereto. In an embodiment, one of the plurality of devices 4000 may be the hub device 1000. Additionally, the hub device 1000 may include a first NLU model, a second NLU model, and an NLG model. In this case, the hub device 1000 may receive a voice input from a user through a microphone or may receive a voice input from at least one of the plurality of devices 4000. When receiving a voice input from a user, the hub device 1000 may process the user voice input by using the ASR model, the first NLU model, the second NLU model, and the NLG model, and may provide a response to the user voice input.

[0404] The hub device 1000 may determine the type of the target device for performing the operation desired by the user based on the received voice signal. The hub device 1000 may receive a voice signal as an analog signal, and may convert the voice portion into computer-readable text by performing automatic speech recognition (ASR). The hub device 1000 may interpret the text by using a first NLU model, and may determine the target device based on the interpretation result. The hub device 1000 may determine at least one of the plurality of devices 4000 as the target device. The hub device 1000 may select a second NLU model corresponding to the determined target device from among a plurality of stored second NLU models. The hub device 1000 may determine the operation to be performed by the target device requested by the user by using the selected second NLU model. When it is determined that there is no second NLU model corresponding to the determined target device among the plurality of stored second NLU models, the hub device 1000 may send at least a portion of the text to at least one of the plurality of devices 4000 or the voice assistant server 2000. The hub device 1000 sends information about the determined operation to the target device so that the determined target device performs the determined operation.

[0405] The hub device 1000 may receive information about the plurality of devices 4000 from the IoT server 3000. The hub device 1000 may determine the target device by using the received information about the plurality of devices 4000. Additionally, the hub device 1000 may send information about the determined operation by using the IoT server 3000 as a relay server to control the target device to perform the determined operation.

[0406] The hub device 1000 may receive a voice input from the user through a microphone, and may send the received voice input to the voice assistant server 2000. In an embodiment, the hub device 1000 may obtain a voice signal from the received voice input, and may send the voice signal to the voice assistant server 2000.

[0407] In ​Among them, the multiple devices 4000 include but are not limited to: a first device 4100 serving as an air conditioner, a second device 4200 serving as a TV, a new device 4300 serving as a washing machine, and a fourth device 4400 serving as a refrigerator. For example, the multiple devices 4000 may include at least one of the following: a smart phone, a tablet personal computer (PC), a mobile phone, a video phone, an e-book reader, a desktop PC, a laptop PC, a netbook computer, a workstation, a server, a personal digital assistant (PDA), a portable multimedia player (PMP), an MP3 player, a mobile medical device, a camera, or a wearable device. In an embodiment, the multiple devices 4000 may be household appliances. The household appliances may include at least one of the following: a TV, a digital video disc (DVD) player, an audio device, a refrigerator, an air conditioner, a vacuum cleaner, an oven, a microwave oven, a washing machine, an air purifier, a set-top box, a home automation control panel, a security control panel, a game console, an electronic key, a camera, or an electronic photo frame.

[0408] The voice assistant server 2000 may determine the type of the target device for performing the operation desired by the user based on the received voice signal. The voice assistant server 2000 may receive the voice signal as an analog signal from the hub device 1000, and may convert the voice part into computer-readable text by performing automatic speech recognition (ASR). The voice assistant server 2000 may interpret the text by using a first NLU model, and may determine the target device based on the interpretation result. Additionally, the voice assistant server 2000 may receive at least part of the text and information about the target device determined by the hub device 1000 from the hub device 1000. In this case, the hub device 1000 converts the voice signal into text by using the ASR model and the first NLU model of the hub device 1000, and determines the target device by interpreting the text. Additionally, the hub device 1000 sends at least part of the text and information about the determined target device to the voice assistant server 2000.

[0409] The voice assistant server 2000 can determine an operation to be performed by a target device for a user request by using a second NLU model corresponding to the determined target device. The voice assistant server 2000 can receive information of multiple devices 4000 from the IoT server 3000. The voice assistant server 2000 can determine the target device by using the received information of the multiple devices 4000. Additionally, the voice assistant server 2000 can control the target device to perform the determined operation by sending information about the determined operation by using the IoT server 3000 as a relay server. The IoT server 3000 can store information about multiple devices 4000 that are connected via a network and pre-registered. In an embodiment, the IoT server 3000 can store at least one of the following: identification information of the multiple devices 4000 (e.g., device ID information), the device type of each of the multiple devices 4000, or information about the function execution ability of each of the multiple devices 4000.

[0410] In an embodiment, the IoT server 3000 can store status information about the power on / off or the operation being performed of each of the multiple devices 4000. The IoT server 3000 can send a control command for performing the determined operation to the target device among the multiple devices 4000. The IoT server 3000 can receive information about the determined target device and information about the determined operation from the voice assistant server 2000, and can send a control command to the target device based on the received information.

[0411] ​ and ​ is a diagram of the voice assistant model 200 that can be executed by the hub device 1000 and the voice assistant server 2000 according to an embodiment.

[0412] Reference ​ and ​ , the voice assistant model 200 is implemented as software. The voice assistant model 200 can be configured to determine a user intention from a user voice input and control a target device related to the user intention. When a device controlled by the voice assistant model 200 is added, the voice assistant model 200 can include a first assistant model 200a and a second assistant model 200b, where the first assistant model 200a is configured to update an existing model to a new model by learning, etc., and the second assistant model 200b is configured to add a model corresponding to the added device to the existing model.

[0413] The first assistant model 200a is a model that determines a target device related to a user's intention by analyzing the user's voice input. The first assistant model 200a may include an ASR model 202, an NLG model 204, a first NLU model 300a, and a device determination model 310. In an embodiment, the device determination model 310 may include the first NLU model 300a. In another embodiment, the device determination model 310 and the first NLU model 300a may be configured as separate elements.

[0414] The device determination model 310 is a model for performing an operation of determining a target device by using the analysis result of the first NLU model 300a. The device determination model 310 may include a plurality of detailed models, and one of the plurality of detailed models may be the first NLU model 300a. The first NLU model 300a or the device determination model 310 may be an AI model.

[0415] When a device controlled by the voice assistant model 200 is added, the first assistant model 200a may learn to update at least the device determination model 310 and the first NLU model 300a. Learning may refer to learning by using training data for training an existing device determination model and a first NLU model and additional training data related to the added device. Additionally, learning may refer to updating the device determination model and the first NLU model by using only the additional training data related to the added device.

[0416] The second assistant model 200b, which is a model dedicated to a specific device, is a model for determining an operation to be performed by a target device corresponding to a user's voice input from among a plurality of operations that can be performed by the specific device. In ​ this case, the second assistant model 200b may include a plurality of second NLU models 300b, an NLG model 206, and an action planning management model 210. The plurality of second NLU models 300b may respectively correspond to a plurality of different devices. The second NLU model, the NLG model, and the action planning management model may be models implemented by a rule-based system. In an embodiment, the second NLU model, the NLG model, and the action planning management model may be AI models. The plurality of second NLU models may be elements of a plurality of function determination models.

[0417] When a device controlled by the voice assistant model 200 is added, the second assistant model 200b can be configured to add a second NLU model corresponding to the added device. That is, in addition to the existing multiple second NLU models 300b, the second assistant model 200b can also include a second NLU model corresponding to the added device. In this case, the second assistant model 200b can be configured to select, from the multiple second NLU models including the added second NLU model, a second NLU model corresponding to the determined target device by using the information about the target device determined by the first assistant model 200a.

[0418] Reference ​ , the second assistant model 200b can include multiple action planning management models and multiple NLG models. In ​ , each of the multiple second NLU models included in the second assistant model 200b can correspond to the ​ second NLU model 300b in, each of the multiple NLG models included in the second assistant model 200b can correspond to the ​ NLG model 206 in, and each of the multiple action planning management models included in the second assistant model 200b can correspond to the ​ action planning management model 210 in.

[0419] In ​ , the multiple action planning management models can be configured to respectively correspond to the multiple second NLU models. Additionally, the multiple NLG models can be configured to respectively correspond to the multiple second NLU models. In another embodiment, one NLG model can be configured to correspond to the multiple second NLU models, and one action planning management model can be configured to correspond to the multiple second NLU models.

[0420] In ​ , when a device controlled by the voice assistant model 200 is added, the second assistant model 200b can be configured to add a second NLU model, an NLG model, and an action planning management model corresponding to the added device.

[0421] In ​In the case where a device controlled by the voice assistant model 200 is added, the first NLU model 300a can be configured to be updated to a new model through learning or the like. Additionally, when the device determines that the model 310 includes the first NLU model 300a, the device determines that the model 310 can be configured such that when a device controlled by the voice assistant model 200 is added, the existing model can be completely updated to a new model through learning or the like. The first NLU model 300a or the device determination model 310 can be an AI model. Learning can refer to learning using the training data for training the existing device determination model and the first NLU model, as well as additional training data related to the added device. Additionally, learning can refer to updating the device determination model and the first NLU model by only using the additional training data related to the added device.

[0422] In ​ In the case where a device controlled by the voice assistant model 200 is added, the second assistant model 200b can be updated by adding the second NLU model, the NLG model, and the action planning management model corresponding to the added device to the existing model. The second NLU model, the NLG model, and the action planning management model can be models implemented through a rule-based system.

[0423] In ​ In the case, the second NLU model, the NLG model, and the action planning management model can be AI models. The second NLU model, the NLG model, and the action planning management model can be managed as one device according to the corresponding device. In this case, the second assistant model 200b can include a plurality of second assistant models 200b-1, 200b-2, and 200b-3 corresponding to a plurality of devices respectively. For example, the second NLU model corresponding to the TV, the NLG model corresponding to the TV, and the action planning management model corresponding to the TV can be managed as the second assistant model 200b-1 corresponding to the TV. Additionally, the second NLU model corresponding to the speaker, the NLG model corresponding to the speaker, and the action planning management model corresponding to the speaker can be managed as the second assistant model 200b-2 corresponding to the speaker. Additionally, the second NLU model corresponding to the refrigerator, the NLG model corresponding to the refrigerator, and the action planning management model corresponding to the refrigerator can be managed as the second assistant model 200b-3 corresponding to the refrigerator.

[0424] When a device controlled by the voice assistant model 200 is added, the second assistant model 200b can be configured to add a second assistant model corresponding to the added device. That is, in addition to the existing multiple second assistant models 200b-1 to 200b-3, the second assistant model 200b can also include a second assistant model corresponding to the added device. In this case, the second assistant model 200b can be configured to select, from the multiple second assistant models including the second assistant model corresponding to the added device, a second assistant model corresponding to the determined target device by using the information about the target device determined by the first assistant model 200a.

[0425] A program executed by the hub device 1000, the voice assistant server 2000, and the multiple devices 4000 according to the present disclosure can be implemented as a hardware component, a software component, and / or a combination of a hardware component and a software component. The program can be executed by any system capable of executing computer-readable instructions.

[0426] Software can include a computer program, code, instructions, or a combination of one or more thereof, and can configure a processing device to operate as needed or command the processing device alone or jointly.

[0427] Software can be implemented in a computer program including instructions stored in a computer-readable storage medium. The computer-readable storage medium can include, for example, a magnetic storage medium (e.g., ROM, RAM, floppy disk, hard disk, etc.) and an optical reading medium (e.g., optical disc, (CD)-ROM, DVD, etc.). The computer-readable recording medium can be distributed in computer systems connected by a network, and can store and execute computer-readable code in a distributed manner. The medium can be computer-readable, storable in a memory, and executable by a processor.

[0428] The computer-readable storage medium can be provided in the form of a non-transitory storage medium. Here, "non-transitory" means that the storage medium does not include a signal and is tangible, but does not distinguish whether the data is stored in the storage medium semi-permanently or temporarily.

[0429] In addition, a program according to an embodiment can be provided in a computer program product. A computer program product is a product that can be purchased between a seller and a buyer.

[0430] The computer program product can include a software program and a computer-readable storage medium in which the software program is stored. For example, the computer program product can include, by a device manufacturer or an e-mall (e.g., Google Play TMSoftware program type products distributed electronically (such as downloadable applications) through, for example, app stores (e.g., store, AppStore, etc.). For electronic distribution, at least a part of the software program can be stored in a storage medium or generated temporarily. In this case, the storage medium can be the storage medium of the manufacturer's server, the server of the electronic mall, or the relay server that temporarily stores the software program.

[0431] In a system including a server and a device, the computer program product can include the storage medium of the server or the storage medium of the device. Alternatively, when there is a third device (such as a smart phone) connected to the server or the device for communication, the computer program product can include the storage medium of the third device. Alternatively, the computer program product can include the software program itself sent from the server to the device or the third device or from the third device to the device.

[0432] In this case, one of the server, the device, and the third device can execute the method according to the embodiment by executing the computer program product. Alternatively, at least two of the server, the device, and the third device can execute the method according to the embodiment in a distributed manner by executing the computer program product.

[0433] For example, a server (such as an IoT server or a voice assistant server, etc.) can execute the computer program product stored in the server and control the device connected to the server for communication to execute the method according to the embodiment.

[0434] As another example, the third device can execute the computer program product and control the device connected to the third device for communication to execute the method according to the embodiment.

[0435] When the third device executes the computer program product, the third device can download the computer program product from the server and execute the downloaded computer program product. Alternatively, the third device can execute the computer program product provided in the free loading state and execute the method according to the embodiment.

[0436] Although the embodiments of the present disclosure have been described through the limited embodiments and the drawings of the present disclosure as described above, various modifications and changes can be made by those of ordinary skill in the art according to the above description. For example, the technology can be executed in a different order from the method and / or with elements different from those of the computer system, module, etc., can be combined or integrated in a different form from the method, or can be replaced or substituted by other elements or equivalents to obtain appropriate results.

Claims

1. A method for a hub device to execute and store a voice assistant model for controlling a device, the method comprising: Receiving information about a first device connected to the hub device, and after receiving the information about the first device, requesting a voice assistant server to update a device determination model stored in the hub device; Receiving the updated device determination model from the voice assistant server and storing the received updated device determination model; Requesting information about a function determination model corresponding to the first device from the voice assistant server; Receiving information about the function determination model corresponding to the first device from the voice assistant server; Monitoring at least one of an average processing speed of a processor of the hub device, a remaining capacity of a memory of the hub device, or an average remaining RAM capacity of the hub device; Determining whether to store the function determination model in the memory of the hub device by comparing the monitored result with the received information about the function determination model corresponding to the first device; Based on determining that the function determination model is stored in the hub device, storing the function determination model corresponding to the first device in the hub device; Selecting at least one candidate hub device from a plurality of devices pre-registered in an Internet of Things (IoT) server according to a user account logged in to the hub device; Selecting one device from the at least one candidate hub device based on usage history information and performance information of each of the at least one candidate hub devices; And Determining the selected one device as a new hub device for replacing the hub device.

2. The method according to claim 1 further comprises: Based on determining that the function determination model is not stored in the hub device, receiving access information about the function determination model corresponding to the first device.

3. The method according to claim 1, wherein Determining whether to store the function determination model in the memory includes: determining whether to store the function determination model based on a resource state of the hub device and information about the function determination model corresponding to the first device.

4. The method according to claim 1, wherein The hub device is selected by the voice assistant server based on resource state information about each of a plurality of devices pre-registered according to a user account.

5. The method according to claim 1, wherein Selecting the at least one candidate hub device includes: selecting the at least one candidate hub device from the plurality of devices pre-registered in the IoT server based on at least one of power supply constancy, computing power, or power consumption amount of each of the plurality of devices pre-registered in the IoT server.

6. The method according to claim 1, wherein, Selecting the one device from the at least one candidate hub device includes: Obtaining usage frequency information about the hub device and each of the at least one candidate hub devices by analyzing a usage history log database stored in the hub device; Obtain speech processing time information about each of the hub device and the at least one candidate hub device by analyzing a performance history log database stored in the hub device; and Select a device for replacing the hub device based on the usage frequency information and the speech processing time information about each of the hub device and the at least one candidate hub device.

7. The method according to claim 1, wherein After receiving information about the first device connected to the hub device, perform an operation of selecting the at least one candidate hub device from the plurality of devices pre-registered in the IoT server.

8. A hub device storing a voice assistant model for controlling a device, the hub device comprising: A communication interface configured to perform data communication with at least one of a plurality of devices, a voice assistant server, or an Internet of Things (IoT) server; A memory configured to store a program including one or more instructions; And A processor configured to execute the one or more instructions of the program stored in the memory to: Receive information about a first device connected to the hub device, After receiving the information about the first device, request the voice assistant server to update a device determination model stored in the memory, and control the communication interface to receive the updated device determination model from the voice assistant server; Store the received updated device determination model in the memory; Request information about a function determination model corresponding to the first device from the voice assistant server, and control the communication interface to receive information about the function determination model corresponding to the first device from the voice assistant server; Monitor at least one of an average processing speed of the processor, a remaining capacity of the memory, or an average remaining RAM capacity of the hub device; Determine whether to store the function determination model in the memory by comparing the monitored result with the received information about the function determination model corresponding to the first device; Based on determining that the function determination model is stored in the hub device, store the function determination model corresponding to the first device in the memory; Select at least one candidate hub device from a plurality of devices pre-registered in the IoT server according to a user account logged in to the hub device; Select one device from the at least one candidate hub device based on usage history information and performance information of each of the at least one candidate hub device; And Determine the selected one device as a new hub device for replacing the hub device.

9. The hub device according to claim 8, wherein, The processor is further configured to execute the one or more instructions to: based on determining that the function determination model is not stored in the hub device, control the communication interface to receive access information about the function determination model corresponding to the first device.

10. The hub device according to claim 8, wherein, The processor is further configured to execute the one or more instructions to: determine whether to store the function determination model in the memory based on the resource state of the hub device and information about the function determination model corresponding to the first device.

11. The hub device according to claim 8, wherein, The hub device is selected by the voice assistant server based on the resource state information about each of the plurality of devices pre-registered according to the user account.

12. The hub device according to claim 8, wherein, The processor is further configured to execute the one or more instructions to: select the at least one candidate hub device from the plurality of devices pre-registered in the IoT server based on at least one of power supply constancy, computing power, or power consumption amount of each of the plurality of devices pre-registered in the IoT server.

13. The hub device according to claim 8, further comprising: a usage history log database that stores usage frequency information about each of the hub device and the at least one candidate hub device; and a performance history log database that stores speech processing time information about each of the hub device and the at least one candidate hub device.

Images