Method and system for controlling plurality of devices by means of voice-based artificial intelligence agent

Abstract

The present invention relates to a method and a system for controlling a plurality of secondary devices by determining, from among a plurality of artificial intelligence models stored in a computing system, an optimal artificial intelligence model for performing a voice-based user request.

Description

Method and system for controlling multiple devices using a voice-based artificial intelligence agent

[0001] The present invention relates to a method and system for controlling a plurality of secondary devices by determining an optimal artificial intelligence model for performing a voice-based user request among a plurality of artificial intelligence models stored in a computing system.

[0002] Typically, artificial intelligence (AI) is implemented through multiple AI models and deep learning based on them.

[0003] These artificial intelligences are being developed to provide various services by taking into account the user's context (e.g., context, environment, and / or intention).

[0004] However, when trying to process a specific task based on large amounts of data, there is a limitation in that the computational cost and time required are significant.

[0005] Due to this, there are certain limitations to the use of AI models in the on-device environment, which has recently been attracting attention.

[0006] On-device AI can refer to artificial intelligence that can process information on a mobile device itself, without the need for an online connection. Recently, on-device AI has been utilized in various smart home appliances, such as refrigerators, water purifiers, and TVs, in conjunction with voice assistants.

[0007] Limitations of these on-devices include the fact that, when an AI voice recognition service is run on a server basis, large-capacity and high-performance services are possible with the server's computing resources, but delays occur depending on the diversity of network resources, and the cost of building and maintaining the server is high.

[0008] In addition, even when configuring an embedded-based artificial intelligence service, there are limitations in coverage and performance compared to large-capacity and high-performance servers, and even when implementing only the functions frequently used by users on an embedded basis and implementing them immediately without consuming network or server resources, there are also problems such as the functions frequently used by each user being different, making it impossible to implement them uniformly on an embedded basis.

[0009] Conversely, if you try to run an AI voice recognition service based on an application that operates offline in an on-device environment, you may encounter problems such as excessive increases in development costs and storage space, as you will need to implement and use a separate AI model for each application that operates offline.

[0010] One embodiment of the present invention has been devised to solve the problems described above, and the purpose is to implement a method and system for controlling multiple devices through a voice-based artificial intelligence agent, in which a master model of a master device controlling multiple secondary devices determines a machine learning model optimized for a user's request obtained through voice recognition, and determines whether a request performed through the determined machine learning model has been performed appropriately and updates the machine learning model as necessary.

[0011] However, the technical problems to be solved by the present invention and embodiments of the present invention are not limited to the technical problems described above, and other technical problems may exist.

[0012] A method for controlling multiple devices through a voice-based artificial intelligence agent according to an embodiment of the present invention is a method for controlling multiple devices through a voice-based artificial intelligence agent by a computing system including a memory and a processor, the method comprising: detecting a user voice by executing a secondary device control service based on a master model that controls the computing system; detecting a query included in the detected voice; determining a first machine learning model that is a secondary model that will process the detected query; controlling the determined first machine learning model to perform the query; examining output data by comparing a first target pair included in the detected query with a second target pair that performed the query; and controlling the first machine learning model to be updated based on a result of the examination of the output data.

[0013] In addition, the step of detecting the user voice includes a step of detecting the user voice based on a voice sensor of a master device or at least one secondary device, a step of obtaining semantic information about the detected user voice based on a multi-modal model, and a step of determining a domain to be used for performing the query based on the obtained semantic information.

[0014] In addition, the step of detecting a query included in the detected voice includes a step of determining a target pair including a target device and a target action based on the determined domain, and a step of searching for a first machine learning model optimized for the determined target pair among a plurality of secondary models previously stored in the computing system.

[0015] In addition, the target device is at least one secondary device that is to perform the query, the target action is an operation that the target device is instructed to process, and the query includes a command that instructs the target device to process the target action.

[0016] In addition, the Master Model includes at least one of a Gating Network, which is an artificial intelligence module that performs task assignment for each of at least one machine learning models included in the computing system, and an Orchestrator, which is an artificial intelligence module that controls the Router.

[0017] In addition, the secondary model is a specialized model whose model characteristic information is matched to the machine learning model on which MoE learning was performed, and is classified into one of an on-device model, a cloud model, and an external model.

[0018] In addition, the step of determining the first machine learning model to process the detected query includes a step of temporarily linking with an external network when the first machine learning model is not determined on the on-device model and the cloud model, a step of determining the first external model searched on the external network as the first machine learning model, and a step of downloading the first external model into the computing system and storing it as one of the on-device model and the cloud model.

[0019] In addition, the step of controlling the determined first machine learning model to perform the query includes the step of obtaining output data output from the first machine learning model that performed the query, wherein the output data includes a second target pair including a target device that performed the query and a target action performed by the target device.

[0020] In addition, the step of reviewing the output data includes a step of determining normal operation based on preset normal operation determination criteria based on at least one or more factors among whether the target pair is normally designated, the time required to perform the query, and the number of queries to perform the query, and if it is determined to be normal operation, it includes at least one step among a step of determining that the first machine learning model continues to perform the query, and a step of controlling the first machine learning model to be updated if it is determined to be abnormal operation.

[0021] In addition, the step of controlling to update the first machine learning model includes the step of generating a query to include a target pair that performed the output data determined to be the abnormal operation, the step of providing the generated query to a user to obtain user feedback on the query, and the step of controlling to update the first machine learning model based on the obtained user feedback.

[0022] In addition, the step of controlling to update the first machine learning model includes at least one of a step of changing a process configuration of the first machine learning model and a step of changing the first machine learning model matched to the query to a second machine learning model.

[0023] Meanwhile, a system for controlling multiple devices through a voice-based artificial intelligence agent according to an embodiment of the present invention comprises: at least one memory; and at least one processor that provides a voice-based artificial intelligence agent that reads out at least one application stored in the memory and controls multiple devices; wherein instructions of the processor include instructions for executing a secondary device control service based on a master model to detect a user voice, detect a query included in the detected voice, determine a first machine learning model as a secondary model to process the detected query, control the determined first machine learning model to perform the query, review output data by comparing a first target pair included in the detected query with a second target pair that performed the query, and update the first machine learning model according to a result of reviewing the output data.

[0024] A method and system for controlling multiple devices using a voice-based artificial intelligence agent according to one embodiment of the present invention has the effect of effectively determining a model optimized for data processing according to a given domain even in an on-device environment, and providing an output according to efficient data processing using the determined model.

[0025] Through this, the method and system for controlling multiple devices through a voice-based artificial intelligence agent according to one embodiment of the present invention can implement and provide an AI agent model that better understands, better executes, and better responds to a given task in any environment, and based on this, it has the effect of significantly improving the quality of service and its performance by enabling easy user-customized control of various types of smart home appliances located in one space.

[0026] However, the effects that can be obtained from the present invention are not limited to the effects mentioned above, and other effects that are not mentioned can be clearly understood from the description below.

[0027] FIG. 1 illustrates an example block diagram of a computing system implementing a secondary device control service according to an embodiment of the present invention.

[0028] FIG. 2 illustrates an example block diagram of a computing device implementing a secondary device control service according to one embodiment of the present invention.

[0029] FIG. 3 illustrates an example block diagram of another aspect of a computing device implementing a secondary device control service according to one embodiment of the present invention.

[0030] FIG. 4 illustrates an internal block diagram of an AI agent model according to one embodiment of the present invention.

[0031] Figure 5 is an example for explaining an external model according to an embodiment of the present invention.

[0032] Figure 6 is a flowchart for explaining a machine learning model modularization method according to an embodiment of the present invention.

[0033] FIG. 8 is a flowchart illustrating a method for controlling multiple devices through a voice-based artificial intelligence agent according to an embodiment of the present invention.

[0034] FIG. 9 is a conceptual diagram illustrating a method for controlling multiple devices through a voice-based artificial intelligence agent according to an embodiment of the present invention.

[0035] The present invention is capable of various modifications and embodiments. Therefore, specific embodiments are illustrated in the drawings and described in detail in the detailed description. The effects and features of the present invention, as well as the methods for achieving them, will become clear with reference to the embodiments described in detail below together with the drawings. However, the present invention is not limited to the embodiments disclosed below and can be implemented in various forms. In the following embodiments, the terms "first," "second," etc. are not used in a limiting sense but are used for the purpose of distinguishing one component from another. Furthermore, the singular expression includes the plural expression unless the context clearly indicates otherwise. Furthermore, terms such as "include" or "have" indicate the presence of a feature or component described in the specification, and do not preemptively exclude the possibility that one or more other features or components may be added. Furthermore, in the drawings, the sizes of components may be exaggerated or reduced for convenience of explanation. For example, the size and thickness of each component shown in the drawings are arbitrarily shown for convenience of explanation, and thus the present invention is not necessarily limited to what is shown.

[0036] Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings. When describing with reference to the drawings, identical or corresponding components are given the same reference numerals and redundant descriptions thereof will be omitted.

[0037]

[0038] FIG. 1 illustrates an example block diagram of a computing system implementing a secondary device control service according to one embodiment of the present invention.

[0039] Referring to FIG. 1, a computing system (1000) implementing a secondary device control service of the present invention includes a user computing device (110), a server computing system (130), and a training computing system (150), and the devices are capable of communicating via a network (170).

[0040] In an embodiment, a user computing device (110) may be divided into a master device (110-M) and / or a secondary device (110-S). The master device (110-M) and a plurality of secondary devices (110-S) have a master-servant relationship, and a master machine learning model (hereinafter, “master model”) that controls the master device (110-M) may operate as an orchestrator that determines, coordinates, and directs a secondary machine learning model (hereinafter, “secondary model”) that controls each of the secondary devices (110-S).

[0041] For example, the master device (110-M) may be a smartphone, and the secondary devices (110-S) may be smart home appliances that operate under the control of the smartphone. That is, the master model of the master device (110-M) can directly / indirectly control the secondary device (110-S) by performing updates that add, change, and delete existing processes that the secondary models include to control the secondary devices (110-S).

[0042] A secondary device control service (hereinafter, referred to as a secondary device control service) that controls a plurality of secondary devices by determining an optimal artificial intelligence model for performing a voice-based user request among a plurality of artificial intelligence models stored in a computing system according to an embodiment of the present invention may be 1) implemented and provided locally by a user computing device (110), 2) implemented and provided in the form of a web service by a server computing system (130) communicating with the user computing device (110), or 3) implemented and provided by the user computing device (110) and the server computing system (130) in connection with each other.

[0043] At this time, in the embodiment, the user computing device (110) and / or the server computing system (130) can train the machine learning model (120 and / or 140) through interaction with a training computing system (150) communicatively connected via a network (170). The training computing system (150) may be separate from the server computing system (130) or may be part of the server computing system (130).

[0044] And at this time, the artificial intelligence model can be 1) trained directly locally by the user computing device (110), 2) trained by the server computing system (130) and the user computing device (110) interacting with each other through a network (170), and 3) trained by a separate training computing system (150) using various training techniques and learning techniques. And the artificial intelligence model trained by the training computing system (150) can be implemented in a manner of being provided / updated by transmitting the model to the user computing device (110) and / or the server computing system (130) through a network (170).

[0045] In some embodiments, the training computing system (150) may be part of a server computing system (130) or part of a user computing device (110).

[0046] The user computing device (110) may include any type of computing device, such as a smart phone, a mobile phone, a digital broadcasting device, a personal digital assistant (PDA), a portable multimedia player (PMP), a desktop, a wearable device, an embedded computing device, and / or a tablet PC.

[0047] Such a user computing device (110) includes at least one processor (111) and memory (112). Here, the processor (111) may be composed of at least one or a plurality of electrically connected processors among a central processing unit (CPU), a graphics processing unit (GPU), application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, micro-controllers, microprocessors, and / or other electrical units for performing functions.

[0048] The memory (112) may include one or more non-transitory / transitory computer-readable storage media such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, and combinations thereof, and may include web storage of a server that performs a memory storage function on the Internet. The memory (112) may store data (113) and commands (114) necessary for the at least one processor (111) to perform functional operations such as training an artificial intelligence model or executing a secondary device control service through an artificial intelligence model.

[0049] In one embodiment, the user computing device (110) may store at least one machine learning model (120).

[0050] In detail, the machine learning model (120) may be various machine learning models such as multiple neural networks (e.g., deep neural networks) or other types of machine learning models including nonlinear models and / or linear models, and may be composed of a combination thereof.

[0051] At this time, the neural network may include at least one of feed-forward neural networks, recurrent neural networks (e.g., long short-term memory recurrent neural networks), convolutional neural networks, and / or other types of neural networks.

[0052] In one embodiment, a user computing device (110) may receive at least one machine learning model (120) from a server computing system (130) via a network (170), store the model in a memory (112), and then execute the stored machine learning model (120) by a processor (111) to perform secondary device control services, etc.

[0053] In another embodiment, the server computing system (130) may include at least one machine learning model (140), perform operations through the machine learning model (140), and communicate data related thereto with the user computing device (110) to provide a secondary device control service to the user.

[0054] For example, a user computing device (110) can perform a secondary device control service in a manner that allows a server computing system (130) to provide output for a user's input using a machine learning model (140) via the web.

[0055] Additionally, the artificial intelligence model may be implemented in such a way that at least some of the machine learning models (120 and / or 140) are executed on a user computing device (110) and the rest are executed on a server computing system (130).

[0056] In addition, the user computing device (110) may include at least one input component (121) that detects a user's input. For example, the user input component (121) may include a touch sensor (e.g., a touch screen and / or a touch pad, etc.) that detects a touch of a user's input medium (e.g., a finger or a stylus), an image sensor that detects a user's motion input, a microphone, a button, a mouse, and / or a keyboard that detects a user's voice input, etc. In addition, the user input component (121) may include an interface and an external controller when receiving an input to an external controller (e.g., a mouse and / or a keyboard, etc.) through an interface.

[0057] The server computing system (130) includes at least one processor (131) and memory (132). Here, the processor (131) may be composed of at least one or a plurality of electrically connected processors among a central processing unit (CPU), a graphics processing unit (GPU), application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, micro-controllers, microprocessors, and / or other electrical units for performing functions.

[0058] And the memory (132) may include one or more non-transitory / transitory computer-readable storage media such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. This memory (132) may store data (133) and commands (134) necessary for the processor (131) to perform functional operations such as training an artificial intelligence model or executing a secondary device control service through an artificial intelligence model.

[0059] In one embodiment, the server computing system (130) may be implemented to include at least one computing device. For example, the server computing system (130) may be implemented to operate multiple computing devices according to a sequential computing architecture, a parallel computing architecture, or a combination thereof. Additionally, the server computing system (130) may include multiple computing devices connected via a network (170).

[0060] Additionally, the server computing system (130) may store at least one machine learning model (140). For example, the server computing system (130) may include a neural network and / or other multi-layer nonlinear model as the machine learning model (140). Exemplary neural networks may include feedforward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks.

[0061] The training computing system (150) includes at least one processor (151) and memory (152). Here, the processor (151) may be composed of at least one or a plurality of electrically connected processors among a central processing unit (CPU), a graphics processing unit (GPU), application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, micro-controllers, microprocessors, and / or other electrical units for performing functions.

[0062] And the memory (152) may include one or more non-transitory / transitory computer-readable storage media such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. This memory (152) may store data (153) and instructions (154) necessary for the processor (151) to perform learning of an artificial intelligence model, etc.

[0063] For example, the training computing system (150) may include a model trainer (160) that trains a machine learning model (120 and / or 140) stored in the user computing device (110) and / or server computing system (130) using various training or learning techniques, such as backpropagation of errors (according to the framework illustrated in FIG. 3).

[0064] For example, such a model trainer (160) may perform updates to one or more parameters of a machine learning model (120 and / or 140) in a backpropagation manner based on a defined loss function.

[0065] In some implementations, performing backward propagation of errors may include performing truncated backpropagation through time. The model trainer (160) may perform a number of generalization techniques (e.g., weight reduction, dropout, and / or knowledge distillation) to improve the generalization ability of the trained machine learning model (120 and / or 140).

[0066] In particular, the model trainer (160) can train the machine learning model (120 and / or 140) based on a series of training data (161). Here, the training data (161) can include data in different formats, such as, for example, images, audio samples, and / or text. Examples of image types that can be used can include video frames, LiDAR point clouds, X-ray images, computed tomography scans, hyperspectral images, and / or various other forms of images.

[0067] Such training data (161) may be provided by a user computing device (110) and / or a server computing system (130). When the training computing device trains a machine learning model (120 and / or 140) on specific data of the user computing device (110), the machine learning model (120 and / or 140) may be characterized as a personalized model.

[0068] And the model trainer (160) includes computer logic utilized to provide the desired function.

[0069] Additionally, the model trainer (160) may be implemented as hardware, firmware, and / or software that controls a general-purpose processor. In one implementation, the model trainer (160) includes a program file stored in a storage device, which may be loaded into a memory (152) and executed by one or more processors (151). In another implementation, the model trainer (160) includes one or more sets of computer-executable data (153) and instructions (154) stored in a tangible computer-readable storage medium, such as a RAM hard disk or an optical or magnetic medium.

[0070] The network (170) includes, but is not limited to, a 3rd Generation Partnership Project (3GPP) network, a Long Term Evolution (LTE) network, a World Interoperability for Microwave Access (WIMAX) network, the Internet, a Local Area Network (LAN), a Wireless Local Area Network (Wireless LAN), a Wide Area Network (WAN), a Personal Area Network (PAN), a Bluetooth network, a satellite broadcasting network, an analog broadcasting network, and / or a Digital Multimedia Broadcasting (DMB) network.

[0071] In general, communication over a network (170) may be performed using any type of wired and / or wireless connection, using various communication protocols (e.g., TCP / IP, HTTP, SMTP, and / or FTP), encodings or formats (e.g., HTML and / or XML), and / or protection schemes (e.g., VPN, Secure HTTP, and / or SSL).

[0072] FIG. 2 illustrates an example block diagram of a computing device implementing a secondary device control service according to one embodiment of the present invention.

[0073] As shown in FIG. 2, the computing device (100) included in the user computing device (110), the server computing system (130), and the training computing system (150) includes a plurality of applications (e.g., Application 1 to Application N). Each application may include a machine learning library and one or more machine learning models. For example, the applications may include an image processing (e.g., Detection, Classification, and / or Segmentation, etc.) application, a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, and / or a chat-bot application.

[0074] In an embodiment, the computing device (100) may include a model trainer (160) for training an artificial intelligence model, and may store and operate the trained artificial intelligence model to provide output data according to predetermined input data (e.g., text, voice, image, video, and / or sensing data based on a specific sensor, etc.).

[0075] Each application of the computing device (100) may communicate with a number of other components of the computing device (100), such as, for example, one or more sensors, a context manager, a device state component, and / or additional components. In one embodiment, each application may communicate with each device component using an API (e.g., a public API). In one embodiment, the API used by each application may be specific to that application.

[0076] FIG. 3 illustrates an example block diagram of another aspect of a computing device implementing a secondary device control service according to one embodiment of the present invention.

[0077] Referring to FIG. 3, a computing device (200) includes a plurality of applications (e.g., Application 1 to Application N). Each application may communicate with a central intelligence layer. For example, the applications may include an image processing application, a text messaging application, an email application, a dictation application, a virtual keyboard application, and / or a browser application. In one embodiment, each application may communicate with the central intelligence layer (and models stored therein) using an API (e.g., a common API across all applications).

[0078] The central intelligence layer may include multiple machine learning models. For example, as illustrated in FIG. 3, at least some of the machine learning models may be provided to each application and managed by the central intelligence layer. In other implementations, two or more applications may share a single machine learning model. For example, in some implementations, the central intelligence layer may provide a single model to all applications. In some implementations, the central intelligence layer may be included within the operating system of the computing device (200) or implemented differently.

[0079] The central intelligence layer may communicate with a central device data layer. The central device data layer may be a centralized data repository for the computing device (200). As illustrated in FIG. 3, the central device data layer may communicate with a number of other components of the computing device (200), such as, for example, one or more sensors, a context manager, a device state component, and / or additional components. In some implementations, the central device data layer may communicate with each device component using an API (e.g., a private API).

[0080] The techniques described herein may refer to servers, databases, software applications, and other computer-based systems, as well as actions taken and information transmitted to or from such systems. It will be appreciated that the inherent flexibility of computer-based systems allows for a wide range of possible configurations, combinations, and division of labor and functionality between and among components. For example, the processes described herein may be implemented using a single device or component, or multiple devices or components operating in combination. Databases and applications may be implemented on a single system or in a distributed system across multiple systems. Distributed components may operate sequentially or in parallel.

[0081]

[0082] -AI Agent Specialization Model (AIAM: Artificial Intelligence Agent Specialization Model)

[0083] In another aspect, the computing system (1000) as above may include an AI agent specialization model (AIAM) according to an embodiment of the present invention.

[0084] Here, the AI ​​agent specialized model (AIAM) according to the embodiment is an AI agent model that determines and uses a machine learning model existing in an on-device environment, a cloud environment, and / or an external environment to obtain and perform a voice-based user request according to the embodiment of the present invention, and may be an artificial intelligence model that includes a data processing algorithm that can act autonomously in a specific environment, solve a task, and achieve a goal.

[0085] These AI agent specialized models (AIAM) may include data processing algorithms to implement cognitive abilities to collect and interpret data from a given environment, decision mechanisms to determine optimal actions based on collected data, execution abilities to execute the determined actions, and learning abilities to improve actions through experience.

[0086] In an embodiment, an AI agent specialized model (AIAM) may acquire certain input data (e.g., not only text, but also voice, image, video, and / or sensing data based on a specific sensor, etc.), and provide output data (e.g., response data for a specific query and / or control signals according to a specific command, etc.) by performing a certain task based on the acquired input data.

[0087] FIG. 4 illustrates an internal block diagram of an AI agent model according to one embodiment of the present invention.

[0088] In detail, referring to FIG. 4, the AI ​​agent model according to the embodiment may include at least one router (RT: Router, Gating Network), an orchestrator (OCT: Orchestrator), an on-device model (OM: On-device Model), a cloud model (CM: Cloud Model), and / or an external model (EM: External Model).

[0089] At this time, in Fig. 4, the AI ​​agent model is described as including the components described above to prevent blurring of features according to an embodiment of the present invention.

[0090] However, it is obvious that a person skilled in the art can understand that, depending on the embodiment, other general components may be included in addition to the components illustrated in FIG. 4, or some of the components illustrated in FIG. 4 may be omitted.

[0091] In more detail, a router (RT: Router, Gating Network) according to an embodiment of the present invention may be an artificial intelligence module that performs task assignment and / or traffic adjustment for multiple models within an AI Agent Specialization Model (AIAM).

[0092] Specifically, the router (RT) can analyze given input data and / or request tasks, etc., to determine which model is most suitable for processing the data.

[0093] At this time, the router (RT) can determine a model optimized for processing given data based on the performance, expertise, and / or previous experience of each model.

[0094] Additionally, the router (RT) can support efficient data processing by distributing a given task to at least one model, taking into account the system load.

[0095] Additionally, the router (RT) can dynamically respond to real-time system changes and adjust tasks assigned to specific models.

[0096] In an embodiment, such a router (RT) may be an artificial intelligence module that selectively determines a model to execute data processing operations optimized for a given domain and / or query.

[0097] That is, in the embodiment, the router (RT) may be an artificial intelligence module that selects a model (hereinafter, domain-specific model) that is judged to perform data processing (e.g., deep learning, etc.) most suitable for a given query among multiple models included in the AI ​​agent specialized model (AIAM).

[0098] For reference, a domain according to an embodiment may refer to data, rules, terminology, problem definitions, and / or processes that a given AI system uses to perform a specific task.

[0099] In an embodiment, a router (RT) may perform data analysis based on characteristics of certain input data (e.g., user input and / or specific sensing data, etc.) and / or requested tasks, and based on this, determine data processing characteristics optimized for the task, detect a certain model implementing the same, and determine a domain-specific model.

[0100] In other words, the router (RT) according to the embodiment may be an artificial intelligence module that detects a model that can most effectively perform data processing according to a given domain and / or query, allocates / distributes the corresponding task processing work, and manages it.

[0101] At this time, the router (RT) according to the embodiment may include a router (RT) previously learned according to a disclosed predetermined algorithm, a router (RT) additionally learned according to the embodiment of the present invention, and / or a router (RT) newly learned in a new manner. A detailed description thereof will be provided later.

[0102] Meanwhile, an orchestrator (OCT) according to an embodiment of the present invention may be an artificial intelligence module that comprehensively controls and manages the overall configuration of an AI agent specialization model (AIAM).

[0103] In detail, in an embodiment, an orchestrator (OCT) can assign various tasks occurring in the entire system to appropriate resources (e.g., a router (RT) and / or a predetermined model, etc.).

[0104] Additionally, the orchestrator (OCT) can manage resources such as available models and hardware resources (e.g., CPU and / or GPU) to ensure efficient use.

[0105] Additionally, the orchestrator (OCT) can monitor the performance of the entire system and adjust specific parameters or optimize network configurations, etc. as needed.

[0106] Additionally, the orchestrator (OCT) can manage the interconnection between multiple routers (RTs) and / or models and control the data flow and processing process.

[0107] That is, in the embodiment, the orchestrator (OCT) performs control and management of the entire system of the AI ​​Agent Specialization Model (AIAM) and can perform the role of a main router (RT) that controls at least one router (RT).

[0108] At this time, the orchestrator (OCT) and the router (RT) according to the embodiment can be closely linked to each other to support efficient operation of a computing system that implements a secondary device control service.

[0109] Specifically, the orchestrator (OCT) monitors the performance of the router (RT) as the manager of the entire system and can adjust the strategy of the router (RT) as needed.

[0110] On the other hand, the router (RT) can implement efficient system control by actually performing the allocation of data processing tasks according to the instructions of the orchestrator (OCT) and / or its own algorithm.

[0111] In an embodiment, the orchestrator (OCT) and / or router (RT) as described above may be a master model (P) that can control and manage the remaining components of the overall system and / or AI agent models (i.e., on-device models, cloud models, external models, etc.).

[0112] Meanwhile, a secondary model (S: Secondary Model) that performs a specific task according to the control and management of a master model (P) (i.e., an orchestrator (OCT) and / or a router (RT), etc.) according to an embodiment of the present invention may be classified into an on-device model (OM: On-device Model), a cloud model (CM: Cloud Model), and / or an external model (EM: External Model).

[0113] In more detail, the on-device model (OM) according to an embodiment of the present invention may be an artificial intelligence model that performs machine learning only within its own computing device and / or system without being connected to a network such as a server or cloud.

[0114] Additionally, a cloud model (CM) according to an embodiment of the present invention may be an artificial intelligence model that performs machine learning and is stored in a cloud used by a computing device and / or system. In the embodiment, the cloud used by the cloud model is described as an internal cloud.

[0115] In addition, the external model (EM) according to an embodiment of the present invention may be a predetermined artificial intelligence model implemented online according to various disclosed algorithms.

[0116] For example, external models (EMs) may include ChatGPT, Gemini, and / or Llama.

[0117] In embodiments, such external models (EMs) may be optionally used as needed to support processing of a given task.

[0118] Figure 5 is an example for explaining an external model according to an embodiment of the present invention.

[0119] Referring to FIG. 5, in the embodiment, the external model (EM) can be used by connecting to the network once or by downloading and installing an already implemented specialized AI model.

[0120] That is, the external model (EM) may function as an on-device model (OM) and / or a cloud model (CM) if the download is completed and stored within the computing device and / or computing system (1000).

[0121] The master model (P) and / or secondary model (S) included in the AI ​​agent specialization model (AIAM) may be an artificial intelligence module implemented based on at least one of a multimodal model, a large language model (LLM), a small large language model (sLLM), a Moe model, and / or a specialized model (SM).

[0122] A multimodal model can refer to a machine learning model that can process various types of data, such as text, images, videos, and voice.

[0123] That is, if the AI ​​agent specialization model (AIAM) according to the embodiment performs a request based on a multimodal model, it is more advanced than the language model that performs natural language understanding and can understand other types of data directly without converting them into text, so it has the effect of quickly performing more complex and difficult user requests.

[0124] sLLM (small Large Language Model) may be an artificial intelligence module implemented as a lightweight version of LLM (Large Language Model).

[0125] That is, if the AI ​​agent specialization model (AIAM) according to the embodiment performs a request based on sLLM, it has the effect of implementing performance similar to that of a large model such as LLM with fewer resources.

[0126] MoE Model (NM: MoE Model) may refer to a given MoE model implemented according to a disclosed universal method.

[0127] For example, a general MoE model (NM) may include Switch Transformer, Conditional Computation in Neural Networks, Sparse Mixture of Experts, and / or Megatron-LM.

[0128] A machine learning model can refer to an artificial intelligence model that has undergone optimization learning for a specific purpose.

[0129] In other words, a machine learning model can be an AI model trained using specialized training data and methods to achieve a specific goal. It can also be an AI model trained for a specific process specific to a specific device.

[0130] In an embodiment, an on-device model (OM), a cloud model (CM), and / or an external model (EM) implemented based on at least one of the multimodal model, LLM, sLLM, Moe model, and / or expert model as described above may be a secondary model (S) that can perform a specific task under the control and management of a master model (P) of an AI agent model (i.e., an orchestrator (OCT) and / or a router (RT), etc.).

[0131] Meanwhile, when the various machine learning models included in the secondary model (S) are later determined as targets of operation by the master model (P), there may be cases where it is difficult to distinguish or determine which domain they are specialized in.

[0132] Due to this, certain limitations may arise in selecting and utilizing the machine learning model most optimized for a specific task. To resolve this, in the embodiment, the computing system (1000) may be modularized by specifying the role and / or function of the secondary model (S).

[0133]

[0134] - How to modularize machine learning models

[0135] Hereinafter, a machine learning model modularization method for implementing modularization for a secondary model (S) in a computing system (1000) according to an embodiment of the present invention will be described in detail with reference to the attached drawings.

[0136] FIG. 6 is a flowchart for explaining a machine learning model modularization method according to an embodiment of the present invention, and FIG. 7 is an example of model characteristic information according to an embodiment of the present invention.

[0137] Referring to FIGS. 6 and 7, in an embodiment, the computing system (1000) can perform Moe learning on machine learning models existing within the computing system (1000). (S101)

[0138] In detail, in an embodiment, the computing system (1000) can perform Moe learning by combining machine learning models existing within the computing system (1000) and a router (RT).

[0139] Accordingly, in the embodiment, the computing system (1000) can learn and build multiple machine learning models included in the AI ​​Agent Specialization Model (AIAM). In this case, the learned machine learning models are described based on whether they are on-device models (OMs), cloud models (CMs), and / or external models (EMs) stored within the computing system (1000).

[0140] That is, the plurality of learned machine learning models may be artificial intelligence models that have undergone optimized learning for a specific device and / or action, and may be artificial intelligence models learned according to training data and methods specialized for the device and / or action.

[0141] In an embodiment, such a machine learning model may be a secondary model (S), which may be at least one of an on-device model (OM), a cloud model (CM), and / or an external model (EM).

[0142] Additionally, in the embodiment, the computing system (1000) can obtain model characteristic information according to learning. (S103)

[0143] Here, model characteristic information according to the embodiment may mean information that specifies the role and / or function of the machine learning model.

[0144] To obtain such model characteristic information, in an embodiment, the computing system (1000) may utilize a model characterization module (MSM) that outputs characteristic information corresponding to a given machine learning model.

[0145] Specifically, in the embodiment, the model specification module (MSM) can monitor and track the task allocation status of the router (RT) for each machine learning model when the aforementioned MoE learning is performed.

[0146] That is, in the embodiment, the model specification module (MSM) can determine which tasks the router (RT) distributes and assigns to which machine learning model as the MoELM learns and operates.

[0147] Depending on the embodiment, the model specification module (MSM) may also manage matching by creating a tag that specifies each task assignment status being tracked.

[0148] Through this, the model specialization module (MSM) in the embodiment can determine the expertise of each of the multiple machine learning models.

[0149] Additionally, in the embodiment, the model specialization module (MSM) can generate model characteristic information corresponding to each machine learning model based on the judged expertise of each machine learning model.

[0150] Figure 7 illustrates an example of model characteristic information according to one embodiment of the present invention.

[0151] At this time, referring to FIG. 7, the model specification module (MSM) as an embodiment can generate the model feature information (SMFI) described above in at least one of the following formats.

[0152] [Format 1] Model feature information (SMFI) in the form of selecting one of the preset machine learning model roles and / or function-specific categories (e.g., question-answering or device control) based on user input.

[0153] [Format 2] Model feature information (SMFI) in the form of natural language that specifies the role and / or function of a machine learning model.

[0154] [Third Format] Model feature information (SMFI) in the form of specifying the role and / or function of a machine learning model in at least one of the first and second formats, and further defining the input data and output data of the machine learning model.

[0155] Continuing, in the embodiment, the model specification module (MSM) can provide the model characteristic information (SMFI) generated as above as output data to the computing system (1000).

[0156] Thus, in the embodiment, the computing system (1000) can obtain model feature information (SMFI) for each machine learning model through linkage with a model specification module (MSM).

[0157] In addition, a computing system (1000) according to one embodiment of the present invention can create a specialized model (SM) by matching the acquired model characteristic information with the machine learning model on which the Moe learning was performed. (S105)

[0158] Here, the expert model (SM) according to the embodiment of the present invention may mean an independently separated machine learning model with matching predetermined model characteristic information.

[0159] In detail, in the embodiment, the computing system (1000) can match the acquired characteristic information as described above to a corresponding machine learning model.

[0160] Additionally, in the embodiment, the computing system (1000) can independently separate and database machine learning models with matched characteristic information.

[0161] That is, in the embodiment, the computing system (1000) can perform modularization by matching characteristic information corresponding to each machine learning model and storing and managing them separately.

[0162] Thus, the computing system (1000) can generate a specialized model (SM), which is an independently separated machine learning model, while matching the characteristic information.

[0163] In this way, in the embodiment, the computing system (1000) can identify the characteristics of each machine learning model within the computing system (1000) and, by reflecting these characteristics, modularize each machine learning model into a small size that can be reused and shared.

[0164] Accordingly, the computing system (1000) can quickly and efficiently select and screen a machine learning model that implements a data processing process optimized for a specific domain (and / or query) with higher accuracy, and can easily support flexible expansion or reduction of the machine learning model based thereon.

[0165] In addition, the computing system (1000) according to one embodiment of the present invention can generate a domain-specific specialized model by matching a predetermined domain to the generated specialized model. (S107)

[0166] Here, a domain according to an embodiment may mean data, rules, terminology, problem definitions, and / or processes that a given AI system uses to perform a specific task (e.g., a query).

[0167] In detail, in an embodiment, the computing system (1000) may obtain certain input data (e.g., text, voice, image, video, and / or sensing data based on a specific sensor, etc.).

[0168] Additionally, in the embodiment, the computing system (1000) can determine a domain corresponding to the acquired input data.

[0169] At this time, the method by which the computing system (1000) determines the domain for input data in the embodiment can be performed based on various disclosed algorithms capable of performing the same, and the embodiment of the present invention does not limit or restrict the algorithm itself.

[0170] Thus, in the embodiment, the computing system (1000) can create a domain-specific specialized model by acquiring a domain for a task to be processed, matching it to the specialized model, and storing it.

[0171] Accordingly, a domain-specific machine learning model that executes data processing (e.g., deep learning, etc.) operations optimized for a given domain may be stored in the computing system (1000).

[0172] For example, when the computing system (1000) confirms the 'characteristics of a task that outputs response data for a given query data' based on the first domain information, it can detect at least one characteristic information that is specified as a 'role and / or function specialized for query response' among a plurality of characteristic information stored in a database.

[0173] And in the embodiment, the computing system (1000) can determine at least one extracted expert model (SM) as a domain-specific expert model.

[0174] Accordingly, a computing system (1000) according to one embodiment of the present invention can build an AI agent specialized model that stores a domain-specific specialized model. (S109)

[0175] In other words, the computing system (1000) can build an AI agent specialized model that stores a machine learning model that implements data processing optimized for a specific domain by using at least some models (i.e., domain-specific specialized models) among a plurality of machine learning models modularized into small sizes.

[0176] At this time, the domain-specific specialized model may be one of an on-device model (OM), a cloud model (CM), and / or an external model (EM).

[0177] Thus, a computing system (1000) according to one embodiment of the present invention can provide a secondary device control service based on an AI agent specialization model. (S111)

[0178] That is, in the embodiment, the computing system (1000) can provide a secondary device control service that provides output data (e.g., response data for a specific query and / or a control signal according to a specific command) for a predetermined input data (e.g., text, voice, image, video, and / or sensing data based on a specific sensor, etc.) using the AI ​​agent specialization model constructed as described above.

[0179] In other words, the master model (P) of the AI ​​agent specialization model receives a user request, extracts a query, detects characteristic information corresponding to the query, and determines a domain-specific specialized model specialized in the detected characteristic information among the secondary models (S) as a machine learning model for performing the user request. Accordingly, the secondary model (S), which is a domain-specific specialized model, can perform the user request by providing output data under the control of the master model (P).

[0180] As described above, in the embodiment, the computing system (1000) can specify the role and / or function of each machine learning model, and at the same time separate and modularize them to a level where they can be reused and shared, and can utilize this to quickly and flexibly build a customized machine learning model optimized for a specific domain, and can provide predetermined output data according to efficient task processing using the built model.

[0181]

[0182] - A method for controlling multiple devices using a voice-based artificial intelligence agent.

[0183] Hereinafter, a method for controlling a secondary device in which an orchestrator (OCT) and / or a router (RT) of a computing system (1000) according to one embodiment of the present invention determines a machine learning model optimized for a user's request obtained through voice recognition, determines whether a request performed through the determined machine learning model was performed appropriately, and updates the machine learning model as necessary is described in detail with reference to the attached drawings.

[0184] Hereinafter, the method of providing a secondary device control service by an orchestrator (OCT) and / or a router (RT) of a computing system (1000) is abbreviated as being performed by a master device (110-M).

[0185] FIG. 8 is a flowchart for explaining a method for controlling multiple devices through a voice-based artificial intelligence agent according to an embodiment of the present invention, and FIG. 9 is a conceptual diagram for explaining a method for controlling multiple devices through a voice-based artificial intelligence agent according to an embodiment of the present invention.

[0186] Referring to FIGS. 8 and 9, in the embodiment, the master device (110-M) can detect user voice as an AI agent. (S201)

[0187] In detail, in the embodiment, the master device (110-M) can detect the user's voice by executing a secondary device control service based on an AI agent specialization model.

[0188] These secondary device control services can be based on AI agent services.

[0189] For example, AI agent services may include voice assistant services for smartphones (e.g., Google Assistant, Apple Siri, Samsung Bixby, etc.), smart camera services (e.g., Google Pixel's HDR+, Apple's Deep Fusion, etc.), fitness tracker and smartwatch services (e.g., Apple Watch, Fitbit, etc.), autonomous driving services for automobiles (e.g., Tesla's Autopilot, etc.), and / or home security services (e.g., Nest Secure, Ring, etc.).

[0190] Additionally, in the embodiment, the master device (110-M) can command at least one secondary device (110-S) and / or secondary model to process a predetermined process based on a user voice detected based on the AI ​​agent service.

[0191] That is, in the embodiment, the master device (110-M) operates as an AI agent and controls at least one secondary device (110-S) to perform processing, and in the embodiment, the secondary device (110-S) can process a predetermined process according to the command contents of the master model.

[0192] Returning to the example, the master device (110-M) can detect the user's voice based on a predetermined voice sensor.

[0193] At this time, in the embodiment, the master device (110-M) can obtain semantic information about the detected user voice based on the multimodal model.

[0194] In an embodiment, the multimodal model may be a deep learning model based on natural language dialogue processing technology that interprets a user's inputted sentence or word-by-word speech utterance to understand the semantic structure, and accordingly, understands the user's intention and infers the most appropriate system response to obtain semantic information.

[0195] This multimodal model can specify a task based on user speech in the form of at least one of text, voice, image, video, and sensor data. The following examples illustrate the multimodal model based on acquiring voice input data.

[0196] In an embodiment, the multimodal model may include a voice I / O processor for voice input / output between a user and a terminal, a Realtime Transport Protocol (RTP) for transmitting a voice stream, an IP Call Control Server (IP-CCS) for controlling a voice call based on IP, a voice recognition unit (ASR: Automatic Speech Recognizer) for recognizing voice between a user and a terminal, a voice synthesis unit (TTS: Text To Speech) for converting text into voice, and a VoiceXML interpreter (Voice Interpreter) for interpreting voice recognition and voice synthesis data.

[0197] That is, in the embodiment, the master device (110-M) can obtain semantic information by immediately identifying the intent of the speech without converting the user's voice into text based on a multimodal model.

[0198] This semantic information can be combined with speech audio in the speech DB of a multimodal model and stored as a single embedding.

[0199] Additionally, in the embodiment, the master device (110-M) can classify semantic information about the user's voice into a basic type including simple commands or everyday questions and / or an advanced type generating complex commands or predetermined routines.

[0200] At this time, in the embodiment, if the user request is basic, the master device (110-M) can immediately execute the request based on its own pre-built process based on the master model (P). On the other hand, if the user request is advanced, the master device can command execution of the request by building a process for determining which secondary model (S) to operate and how.

[0201] That is, in the embodiment, the master device (110-M) can provide an immediate answer on its own or command the secondary model (S) to process a predetermined process depending on the type of semantic information for the user's voice.

[0202] Additionally, in the embodiment, the master device (110-M) can detect a query included in the detected voice. (S203)

[0203] In detail, in an embodiment, the master device (110-M) can detect pre-stored semantic information and / or the context of the query based on the voice DB of the multimodal model.

[0204] Additionally, a query according to an embodiment may include certain data that can specify a target task of data processing.

[0205] In detail, a query according to an embodiment may include a secondary device (hereinafter, “target device”) that is to perform the query and an operation (hereinafter, “target action”) that is commanded to be processed by the target device.

[0206] Additionally, a query according to an embodiment may be matched in advance with a secondary model that controls the target device to perform the target action.

[0207] For example, the query “Turn on the living room air conditioner” may be matched with the target device “Nth air conditioner (living room air conditioner)” and the target action “Turn on, set the indoor temperature to 26 degrees.” In addition, the first secondary model matched to the query may control the “Nth air conditioner” to “Turn on, set the indoor temperature to 26 degrees.”

[0208] That is, in the embodiment, the master device (110-M) can detect a query for which a secondary model is preset to control the target device to perform a target action from the detected voice.

[0209] Additionally, in the embodiment, the master device (110-M) can determine a domain according to the detected query.

[0210] Here, again, a domain according to an embodiment may mean data, rules, terminology, problem definitions, and / or processes that a given AI system uses to perform a specific task (e.g., a query).

[0211] At this time, in the embodiment, the master device (110-M) can determine the domain (in the embodiment, the target device and / or target action) for the query based on various algorithms capable of determining the domain, and the embodiment of the present invention does not limit or restrict the algorithm itself.

[0212] Meanwhile, if the target device and / or target action do not match the query or are abstract, the target task (i.e., domain) may not be specified. In this case, in the embodiment, the master device (110-M) may execute a target determination process that determines the target device, target action, and / or secondary model for performing the query. The target determination process will be described in detail in step S205 described below.

[0213] Additionally, in the embodiment, the master device (110-M) can determine a first machine learning model corresponding to the detected query. (S205)

[0214] Here, the first machine learning model determined according to the embodiment may mean a model that performs a predetermined task processing according to given input data (a query in the embodiment).

[0215] This first machine learning model may be at least one of the secondary models (S) described above.

[0216] At this time, in other words, the secondary model (S) according to the embodiment may mean a model that can perform a specific task according to the control and management of the master model (P) (i.e., the orchestrator (OCT) and / or router (RT), etc.) responsible for controlling and managing the operation of a given AI system.

[0217] In an embodiment, such secondary model (S) may be at least one of an on-device model (OM), a cloud model (CM), and / or an external model (EM).

[0218] In detail, in an embodiment, the master device (110-M) may determine, if a preset secondary model already exists for the detected query, the secondary model as the first machine learning model.

[0219] Meanwhile, in the embodiment, the master device (110-M) can execute a target determination process to specify a target task when there is no preset secondary model or the secondary model is unspecified for the detected query.

[0220] In an embodiment, the target determination process may include a process of receiving user voice input via a voice recognition interface to determine a target task (i.e., a target device and target action). The voice recognition interface provided in this target determination process may be identical to the voice recognition interface provided when the user's voice is initially detected.

[0221] In the embodiment, the reason for executing the target determination process can be divided into cases where the master device (110-M) 1) does not understand the query itself included in the user voice, 2) the target device included in the query is unspecified, and / or 3) the target action included in the query is unspecified.

[0222] In the case of 1), in the embodiment, the master device (110-M) can request the user to input the user's voice again by asking the user a question using a query understood by the master model.

[0223] In cases 2) and 3), in the embodiment, the master device (110-M) may provide a question for the user to select by listing keywords mapped to the target device and / or device and / or action presumed to be the target action.

[0224] Accordingly, in the embodiment, the master device (110-M) can determine a target device and / or a target action (hereinafter, target pair) based on the target determination process.

[0225] Additionally, in the embodiment, the master device (110-M) can determine a first machine learning model optimized for the determined target device to perform the target action.

[0226] At this time, in the embodiment, the master device (110-M) can detect at least one characteristic information corresponding to the domain information extracted from the query according to the above-described [machine learning model modularization method].

[0227] Additionally, in the embodiment, the master device (110-M) can extract a first machine learning model that matches at least one detected characteristic information.

[0228] That is, the first machine learning model extracted above may be a machine learning model optimized for performing the target pair.

[0229] Meanwhile, the first machine learning model determined above may be one of an on-device model (OM), a cloud model (CM), and / or an external model (EM).

[0230] In detail, in an embodiment, the master device (110-M) may determine an on-device model (OM) that performs machine learning only within its own computing device and / or system without being connected to a network such as a server or cloud, as the first machine learning model.

[0231] In a case where the first machine learning model is determined as an on-device model (OM), a machine learning model matching a query with a similarity higher than a preset standard may exist within a computing device and / or system.

[0232] At this time, the on-device model (OM) can provide advantages such as privacy protection, real-time processing, and reduced dependence on Internet connection because all processing is completed within the device without sending data externally.

[0233] Additionally, in the embodiment, the master device (110-M) may determine a cloud model (CM) that performs machine learning stored in an internal cloud used by the computing device and / or system as the first machine learning model.

[0234] In the case where the above first machine learning model is determined to be a cloud model (CM), it may be the case that a model optimized for the target device to perform the target action does not exist in the on-device model (OM).

[0235] At this time, since the above cloud model (CM) is an internal cloud model, there is no need to connect to an external network like the on-device model, so it has the advantage of enabling rapid process processing.

[0236] That is, in the embodiment, the master device (110-M) can primarily search for and determine an optimization model on the cloud model (CM) before connecting to an external network.

[0237] Additionally, in the embodiment, the master device (110-M) can determine an external model (EM) implemented online according to various disclosed algorithms as the first machine learning model.

[0238] In the case where the above first machine learning model is determined to be an external model (EM), it may be the case where it is determined that the optimization model does not exist in the on-device model (OM) and / or cloud model (CM).

[0239] At this time, in the embodiment, the master device (110-M) can determine the external model (EM) as the first machine learning model by temporarily connecting to an external network to search for an optimization model and connecting to the model once.

[0240] In addition, in the embodiment, the master device (110-M) can download the external model (EM), which is a specialized AI model already implemented in an external network, and store it as an on-device model (OM) and / or a cloud model (CM).

[0241] That is, the external model (EM) may operate as an on-device model (OM) and / or a cloud model (CM) if the download is completed and stored on the own computing device and / or system.

[0242] Additionally, in the embodiment, the master device (110-M) can control the determined first machine learning model to perform a query. (S207)

[0243] At this time, in the embodiment, the master device (110-M) can obtain data output from the first machine learning model that performed the query.

[0244] That is, in the embodiment, the master device (110-M) can generate and provide output data (e.g., response data for a specific query and / or control signals according to a specific command) for input data in the form of a user voice.

[0245] In other words, in the embodiment, the master device (110-M) can perform a predetermined request task based on given input data using the first machine learning model determined as above, and provide output data according to the performed data processing.

[0246] Additionally, in the embodiment, the master device (110-M) can review the output data according to the query execution. (S209)

[0247] In detail, in the embodiment, the master device (110-M) can determine whether the target pair has been performed normally by examining the output data of the target pair that performed the query by the determined first machine learning model.

[0248] Here, the master device (110-M) according to the embodiment can review the output data according to the query execution by comprehensively judging 1) whether the target device is normally designated, 2) whether the target action is normally designated, 3) the time required to perform the query, and / or 4) the number of queries until the query is performed.

[0249] At this time, for 1) and 2), the master device (110-M) can review the output data by comparing the first target pair included in the detected query with the second target pair included in the output data according to the query execution.

[0250] In other words, in the embodiment, the master device (110-M) can predetermine the target pair included in the query when detecting the query as a criterion for determining normal operation.

[0251] Meanwhile, in order to review output data on 3) the time required to perform a query and / or 4) the number of queries to perform a query, the master device (110-M) in the embodiment may set separate normal operation judgment criteria.

[0252] For example, the master device (110-M) can preset a first reference time and / or a first reference query count that serve as criteria for determining normal operation.

[0253] That is, in the embodiment, the master device (110-M) can review the output data by comparing the time required and / or the number of queries included in the output data with the preset first reference time and / or the number of first reference queries.

[0254] At this time, if the target pair of the output data, the required time, and / or the number of queries satisfy the above-described normal operation judgment criteria, the master device (110-M) in the embodiment may determine that the first machine learning model continues to perform the first query. That is, in the embodiment, the master device (110-M) may determine the above case as normal operation.

[0255] Meanwhile, in the embodiment, the master device (110-M) may determine that the case is abnormal operation if the target pair of output data, the required time, and / or the number of queries do not satisfy the above-described normal operation determination criteria.

[0256] Accordingly, in the embodiment, the master device (110-M) may update the machine learning model based on its own process and / or user feedback when the output data is determined to be operating abnormally. (S211)

[0257] In detail, in the embodiment, the master device (110-M) can update the configuration of a process that an existing machine learning model operates to perform a query, or perform an update that changes the first machine learning model matched to the query to a second machine learning model.

[0258] To this end, in the embodiment, the master device (110-M) can generate a query to specify a target device and / or a target action, and obtain user feedback on the generated query.

[0259] At this time, the generated query may be, for example, audio output data generated from the master device (110-M), and the user feedback may be input data in the form of user voice detected from the master device (110-M) and / or the secondary device (110-S).

[0260] In an embodiment, the master device (110-M) can generate a query based on output data determined to be in abnormal operation.

[0261] In detail, in the embodiment, the master device (110-M) can generate a query by including a target device and / or a target action included in the output data, or by changing the target device and / or the target action included in the output data to another device and / or another action according to a predetermined standard set in advance.

[0262] Pre-established criteria for determining which other devices and / or actions are changed here may include, for example, similarity of word pronunciation, similarity of spacing between devices, similarity of values, etc.

[0263] For example, if the master device (110-M) assumes that the output data is a “control signal to turn on the living room air conditioner and determine the temperature to 26 degrees,” it can generate a query such as “Is the air conditioner currently in operation a living room air conditioner?” or “Should I set the air conditioner currently in operation as a bedroom air conditioner?”

[0264] That is, when a user attempts to control a given device based on voice, the procedure for setting a given action through question-and-answer with the device from the beginning is simplified, thereby reducing user fatigue and increasing user satisfaction due to device control through a voice agent.

[0265] Additionally, in the embodiment, the master device (110-M) can obtain user feedback by providing the generated query to the user. In this case, the user feedback may include the user's voice response to the generated query, which determines the target device and / or target action.

[0266] For example, a user can generate and provide user feedback such as, “No, set the air conditioner that is currently running to the master bedroom air conditioner.” or “Okay, but from now on, set the master bedroom air conditioner to 25 degrees.”

[0267] At this time, the device obtaining user feedback may be one of the master device (110-M) and / or multiple secondary devices (110-S).

[0268] Based on such user feedback, in the embodiment, the master device (110-M) may change the process configuration of an existing machine learning model when performing a request according to a query, or perform an update that changes the machine learning model matched to the query.

[0269] Accordingly, a specialized machine learning model is determined to perform user requests, enabling faster and more accurate responses. In addition, even when a user's request is not performed properly, the query process is simplified through customized user queries about the problem without having to go through multiple Q&A sessions with the user.

[0270] Meanwhile, in the embodiment, the master device (110-M) can set a user-customized query that controls at least one secondary device to perform an action desired by the user.

[0271] To this end, in an embodiment, the master device (110-M) can obtain input for setting a user-customized query through a voice recognition interface.

[0272] Additionally, in the embodiment, the master device (110-M) can acquire and determine a target device and / or target action to be operated when acquiring a user-customized query based on a voice recognition interface.

[0273] At this time, there may be multiple target devices and / or target actions corresponding to one user-customized query.

[0274] For example, in a request task set to a user-customized query called “hot,” the first target device and the first target action may be “living room air conditioner” and “power ON, 26 degrees,” and the second target device and the second target action may be “air purifier” and “power OFF.”

[0275] Additionally, in the embodiment, the master device (110-M) may generate a predetermined routine by setting the operating conditions of the corresponding user-customized query.

[0276] For example, the operating condition of a user-customized query may be a preset time (e.g., 7:00 PM) or the location of a preset master device (110-M) (e.g., within 1 km).

[0277] Accordingly, the efficiency of voice commands is increased because two or more secondary devices can be controlled with a single query, and the usability of devices through voice commands is increased because commands frequently used by users are stored and used as user-customized queries.

[0278]

[0279] The embodiments of the present invention described above may be implemented in the form of program commands that can be executed through various computer components and recorded on a computer-readable recording medium. The computer-readable recording medium may include program commands, data files, data structures, etc., either singly or in combination. The program commands recorded on the computer-readable recording medium may be specially designed and configured for the present invention or may be known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specifically configured to store and execute program commands, such as ROMs, RAMs, and flash memories. Examples of program commands include not only machine language codes generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc. Hardware devices may be changed into one or more software modules to perform processing according to the present invention, and vice versa.

[0280] The specific implementations described in the present invention are exemplary embodiments and do not limit the scope of the present invention in any way. For the sake of brevity, descriptions of conventional electronic components, control systems, software, and other functional aspects of the systems may be omitted. In addition, the lines connecting or connecting members between components illustrated in the drawings are merely representative of functional connections and / or physical or circuit connections, and may be replaced or represented as various additional functional connections, physical connections, or circuit connections in an actual device. In addition, unless specifically mentioned as “essential,” “important,” etc., a component may not be absolutely necessary for the application of the present invention.

[0281] Although the detailed description of the present invention has been described with reference to preferred embodiments of the present invention, it will be understood by those skilled in the art or having ordinary knowledge in the art that various modifications and changes can be made to the present invention without departing from the spirit and technical scope of the present invention as set forth in the claims below. Accordingly, the technical scope of the present invention should not be limited to the contents described in the detailed description of the specification, but should be defined by the claims.

[0282] The form for carrying out the invention is the same as the best form for carrying out the invention described above.

[0283] The present invention can effectively determine a model optimized for data processing according to a given domain even in an on-device environment, and provide an output according to efficient data processing through the determined model, thereby implementing and providing an agent model that accomplishes a given task in any environment, and based on this, it can easily control various types of smart home appliances located in one space in a user-customized manner, thereby significantly improving the quality of service and its performance, and thus has industrial applicability.

Claims

1. A method for controlling multiple devices through a voice-based artificial intelligence agent by a computing system including a memory and a processor, A step of detecting a user's voice by executing a secondary device control service based on a master model that controls the above computing system; A step of detecting a query included in the detected voice; A step of determining a first machine learning model, which is a secondary model to process the above-detected query; A step of controlling the above-determined first machine learning model to perform the above-determined query; A step of reviewing output data by comparing the first target pair included in the detected query with the second target pair that performed the query; and A step of controlling to update the first machine learning model according to the review result of the output data; A method for controlling multiple devices using a voice-based artificial intelligence agent.

2. In paragraph 1, The step of detecting the user voice is as follows: A step of detecting a user's voice based on a voice sensor of a master device or at least one secondary device, A step of obtaining semantic information about the detected user voice based on a multi-modal model, A step of determining a domain to be used in performing the query according to the acquired semantic information A method for controlling multiple devices using a voice-based artificial intelligence agent.

3. In paragraph 2, The step of detecting a query included in the above detected voice is as follows: A step of determining a target pair including a target device and a target action based on the determined domain, A step of searching for a first machine learning model optimized for the determined target pair among a plurality of secondary models previously stored in the computing system. A method for controlling multiple devices using a voice-based artificial intelligence agent.

4. In paragraph 3, The above target device is, At least one secondary device that is the target of the above query, The above target action is, An action that is commanded to be processed by the target device, The above query is, A command that includes a command that instructs the target device to process the target action. A method for controlling multiple devices using a voice-based artificial intelligence agent.

5. In paragraph 1, The above Master Model is, A router (Gating Network), which is an artificial intelligence module that performs task assignment for each of at least one machine learning model included in the computing system, and At least one orchestrator, which is an artificial intelligence module that controls the above router, is included. A method for controlling multiple devices using a voice-based artificial intelligence agent.

6. In paragraph 1, The above secondary model is, It is a specialized model whose model characteristic information is matched to the machine learning model on which MoE learning was performed. It is divided into one of the on-device model, cloud model, and external model. A method for controlling multiple devices using a voice-based artificial intelligence agent.

7. In paragraph 6, The step of determining the first machine learning model to process the above-detected query is: When the first machine learning model is not determined on the above on-device model and the above cloud model, a step of temporarily connecting with an external network; A step of determining the first external model searched on the external network as the first machine learning model, A step of downloading the first external model into the computing system and storing it as one of an on-device model and a cloud model. A method for controlling multiple devices using a voice-based artificial intelligence agent.

8. In paragraph 1, The step of controlling the first machine learning model determined above to perform the query comprises: A step of obtaining output data output from a first machine learning model that performed the above query, The above output data is, A second target pair including a target device that performed the query and a target action performed by the target device. A method for controlling multiple devices using a voice-based artificial intelligence agent.

9. In paragraph 1, The step of reviewing the above output data is: It includes a step of determining normal operation according to a preset normal operation determination criterion based on at least one or more factors among whether the target pair is normally designated, the time required to perform the query, and the number of queries to perform the query. If it is determined that the operation is normal, a step of determining that the first machine learning model continues to perform the query; In case it is determined that the operation is abnormal, at least one step of controlling to update the first machine learning model is included. A method for controlling multiple devices using a voice-based artificial intelligence agent.

10. In paragraph 9, The step of controlling to update the above first machine learning model is: A step of generating a query to include the target pair that performed the output data determined to be the above abnormal operation, A step of providing the generated query to the user and obtaining user feedback on the query; A step of controlling to update the first machine learning model based on the acquired user feedback. A method for controlling multiple devices using a voice-based artificial intelligence agent.

11. In paragraph 10, The step of controlling to update the above first machine learning model is: A step of changing the process configuration of the above first machine learning model, and At least one step of changing the first machine learning model matching the above query into a second machine learning model is included. A method for controlling multiple devices using a voice-based artificial intelligence agent.

12. At least one memory; and At least one processor providing a voice-based artificial intelligence agent that controls a plurality of devices by reading out at least one application stored in the memory; The instructions of the above processor are: Detect user voice by executing secondary device control service based on Master Model, Detecting a query included in the above detected voice, Determine the first machine learning model, which is a secondary model that will process the above-detected query, Controlling the above-determined first machine learning model to perform the above query, By comparing the first target pair included in the above-detected query with the second target pair that performed the above-detected query, the output data is reviewed, A command that controls updating the first machine learning model based on the review result of the output data. A system for controlling multiple devices using a voice-based artificial intelligence agent.

Images