Electronic device for supporting performance of function corresponding to utterance of user by using artificial intelligence model, and operation method thereof

Abstract

Provided is an electronic device comprising: a microphone; a communication circuit for communicating with an external electronic device; a memory storing instructions; and a processor. The instructions, when executed by the processor, may cause the electronic device to: acquire an utterance of a user through the microphone or the communication circuit; receive, from at least one external electronic device through the communication circuit, information about at least one control command for controlling the at least one external electronic device; generate a first prompt to be input into a first large language model (LLM), on the basis of the received information and the utterance; identify, from an output value that has been output from the first LLM as a result of inputting the first prompt into the first LLM, an update target in the at least one external electronic device and update content of a second prompt to be input into a second LLM related to the update target; and transmit the update content through the communication circuit to an external electronic device configured to generate the second prompt.

Description

An electronic device that supports the performance of a function corresponding to a user's utterance using an artificial intelligence model, and a method of operation thereof

[0001] The present disclosure relates to an electronic device configured to support the performance of a function corresponding to a user's utterance using an artificial intelligence model, and a method of operating the same.

[0002] An electronic device can recognize segments within an audio signal where a user's utterance is located and recognize the user's intent from the utterance. For example, the electronic device can understand the user's intent by utilizing a natural language understanding model. It can provide the user with an AI (artificial intelligent) agent service (or voice assistant service) that supports the output of a response corresponding to the user's intent or the execution of a function.

[0003] With the advancement of artificial intelligence models (e.g., Large Language Models (LM)), various methods to introduce LLM into device control are being studied. Through such research, optimal device control can be expected by analyzing even ambiguous user utterances in conjunction with surrounding context.

[0004] The information described above is provided as related art for the purpose of aiding understanding of the present disclosure. No claim or determination is made as to whether any of the foregoing may be applied as prior art related to the present disclosure.

[0005] By learning through various conversations, the LLM can respond to user utterances that are not direct voice commands. For example, along with the user's utterance, the LLM may be input as a prompt (or input value), such as a list of devices to be controlled, information regarding functions available on each device, the user's current location, weather, or time. Based on the output value from the LLM, a device can be selected and a function can be performed. This processing method using the LLM enables responses even to ambiguous user utterances. For example, even if a user in the living room says "Ah, it's hot" instead of "Turn on the air conditioner," the LLM can perform the action of turning on the air conditioner located in the living room based on the input prompt.

[0006] LLM may misunderstand the user's intent and present incorrect results when the amount of input information (e.g., tokens) (e.g., the number of devices to be controlled) is large. For instance, a ventilation fan might operate as an incorrect response to a user's utterance, "Ah, it's hot."

[0007] Various embodiments of the present disclosure may provide an electronic device that supports a more accurate response to a user's utterance by utilizing an LLM that has relatively low performance and relatively low cost. Here, the low-performance and low-cost LLM (hereinafter referred to as a tiny or small LLM) may refer, for example, to an LLM that requires a relatively small amount of input information and / or an LLM that has a relatively small number of parameters indicating the scale of said LLM.

[0008] Various embodiments of the present disclosure may provide an electronic device that enables a large-scale LLM to accurately respond to a user's utterance by using a result value output from a small-scale LLM and updating a prompt to be fed into the small-scale LLM based on the output result value.

[0009] Various embodiments of the present disclosure may provide an electronic device that enables a large-scale LLM to respond more accurately to a user's utterance by updating a prompt to be fed into a small-scale LLM based on a result value output by the large-scale LLM using the user's utterance. The technical problems to be solved by the present disclosure are not limited to those mentioned above, and other unmentioned technical problems will be clearly understood by those skilled in the art from the description below.

[0010] According to one embodiment, an electronic device is provided comprising: a microphone; a communication circuit for communicating with an external electronic device; a memory for storing instructions; and at least one processor. When the instructions are executed individually or collectively by the at least one processor, the electronic device may receive information regarding at least one control command from the at least one external electronic device through the communication circuit, which is used to acquire a user's utterance through the microphone or the communication circuit and to control at least one external electronic device; generate a first prompt to be input into a first large language model (LLM) based on the received information and the utterance; determine an update target from the at least one external electronic device and an update content of a second prompt to be input into a second LLM associated with the update target from an output value output from the first LLM as a result of inputting the first prompt into the first LLM; and transmit the update content through the communication circuit to an external electronic device configured to generate the second prompt.

[0011] According to one embodiment, a method for operating an electronic device is provided. The method may include: acquiring a user’s utterance through a microphone of the electronic device or a communication circuit of the electronic device; receiving information regarding at least one control command to control at least one external electronic device from the at least one external electronic device through the communication circuit; generating a first prompt to be input into a first large language model (LLM) based on the received information and the utterance; checking the update content of a second prompt to be inserted into an update target and a second LLM associated with the update target from an output value output from the first LLM as a result of inputting the first prompt into the first LLM from the at least one external electronic device; and transmitting the update content to an external electronic device configured to generate the second prompt through the communication circuit.

[0012] According to one embodiment, an electronic device is provided comprising: a microphone; a communication circuit for communicating with an external electronic device; a memory for storing instructions; and at least one processor. When the instructions are executed individually or collectively by the at least one processor, the electronic device may acquire a user's utterance through the microphone or the communication circuit, generate a first prompt to be input into a first large language model (LLM) based on the utterance and a list of devices registered with the electronic device, check an update target in the list of devices and an update content of a second prompt to be input into a second LLM related to the update target from an output value output from the first LLM as a result of inputting the first prompt into the first LLM, and transmit the update content to an external electronic device configured to generate the second prompt through the communication circuit.

[0013] According to one embodiment, a method for operating an electronic device is provided. The method may include: acquiring a user’s utterance through a microphone of the electronic device or a communication circuit of the electronic device; generating a first prompt to be input into a first large language model (LLM) based on the utterance and a list of devices registered in the electronic device; checking an update target in the list of devices and an update content of a second prompt to be input into a second LLM related to the update target from an output value output from the first LLM as a result of inputting the first prompt into the first LLM; and transmitting the update content to an external electronic device configured to generate the second prompt through the communication circuit.

[0014] According to an embodiment of the present disclosure, an electronic device can respond more accurately to a user's utterance by using an LLM. In addition, various effects that can be identified directly or indirectly through this document may be provided.

[0015] FIG. 1 is a block diagram of an electronic device in a network environment according to various embodiments.

[0016] FIG. 2 illustrates an IoT (internet of things) system according to one embodiment.

[0017] FIG. 3 illustrates an AI agent service system for responding more accurately to a user's utterance according to one embodiment.

[0018] FIG. 4 is a diagram illustrating an operation to update a prompt to be input into a small LLM so as to respond more accurately to a user's utterance in the system of FIG. 3 according to one embodiment.

[0019] FIGS. 5a, 5b, 5c, and 5d are drawings for illustrating an example of an AI agent service according to one embodiment.

[0020] FIG. 6 is a diagram illustrating an operation to update a prompt to be input into a small LLM so as to respond more accurately to a user's utterance in the master device of FIG. 3 according to one embodiment.

[0021] FIG. 7 is a diagram illustrating an example of an AI agent service according to one embodiment.

[0022] FIG. 8 is a flowchart illustrating the operation of an electronic device for updating a prompt according to one embodiment.

[0023] Figure 9 illustrates an example of a message provided to a user following an update of the prompt.

[0024] FIG. 10 is a flowchart illustrating the operation of an electronic device for updating a prompt according to one embodiment.

[0025] FIG. 11 illustrates an example of a message provided to a user following an update of a prompt.

[0026] Hereinafter, embodiments of the present disclosure are described in detail with reference to the drawings so that those skilled in the art can easily practice them. However, the present disclosure may be embodied in various different forms and is not limited to the embodiments described herein. In relation to the description of the drawings, the same or similar reference numerals may be used for identical or similar components. Furthermore, in the drawings and related descriptions, descriptions of well-known functions and configurations may be omitted for clarity and brevity.

[0027] FIG. 1 is a block diagram of an electronic device (101) in a network environment (100) according to various embodiments. Referring to FIG. 1, in the network environment (100), the electronic device (101) may communicate with an electronic device (102) through a first network (198) (e.g., a short-range wireless communication network) or may communicate with at least one of an electronic device (104) or a server (108) through a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) through a server (108). According to one embodiment, the electronic device (101) may include a processor (120), memory (130), input module (150), sound output module (155), display module (160), audio module (170), sensor module (176), interface (177), connection terminal (178), haptic module (179), camera module (180), power management module (188), battery (189), communication module (190), subscriber identification module (196), or antenna module (197). In some embodiments, at least one of these components (e.g., connection terminal (178)) may be omitted from the electronic device (101), or one or more other components may be added. In some embodiments, some of these components (e.g., sensor module (176), camera module (180), or antenna module (197)) may be integrated into a single component (e.g., display module (160)).

[0028] The processor (120) can control at least one other component (e.g., hardware or software component) of the electronic device (101) connected to the processor (120) by executing software (e.g., program (140)), for example, and can perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor (120) can store commands or data received from other components (e.g., sensor module (176) or communication module (190)) in volatile memory (132), process the commands or data stored in volatile memory (132), and store the resulting data in non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., central processing unit or application processor) or an auxiliary processor (123) that can operate independently or together with it (e.g., graphics processing unit, neural processing unit (NPU), image signal processor, sensor hub processor, or communication processor). For example, if the electronic device (101) includes a main processor (121) and an auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a designated function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as part thereof.

[0029] The auxiliary processor (123) may control at least some of the functions or states associated with at least one component of the electronic device (101) (e.g., display module (160), sensor module (176), or communication module (190)) on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. According to one embodiment, the auxiliary processor (123) (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module (180) or communication module (190)). According to one embodiment, the auxiliary processor (123) (e.g., neural network processing unit) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or through a separate server (e.g., server (108)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers.An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may include a software structure, either additionally or substantially.

[0030] The number of processors (120) may be one or more. For example, the processor (120) may have the structure of a multi-core processor such as a dual core, a quad core, or a hexa core.

[0031] The processor (120) can control the operations of the electronic device (101) by executing instructions stored in memory (130). For example, the processor (120) may correspond to a plurality of processors that divide and collectively perform a plurality of operations among the processors.

[0032] The memory (130) can store various data used by at least one component of the electronic device (101) (e.g., processor (120) or sensor module (176)). The data may include, for example, input data or output data for software (e.g., program (140)) and related commands. The memory (130) may include volatile memory (132) or non-volatile memory (134).

[0033] The program (140) may be stored as software in memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0034] The input module (150) can receive commands or data to be used for a component of the electronic device (101) (e.g., processor (120)) from outside the electronic device (101) (e.g., user). The input module (150) may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0035] The sound output module (155) can output a sound signal to the outside of the electronic device (101). The sound output module (155) may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as multimedia playback or recording playback. The receiver may be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part thereof.

[0036] The display module (160) can visually provide information to an external (e.g., user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling said device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of the force generated by said touch.

[0037] The audio module (170) can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150) or output sound through the sound output module (155) or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphones) connected directly or wirelessly to the electronic device (101).

[0038] The sensor module (176) can detect the operating state of the electronic device (101) (e.g., power or temperature) or the external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, the sensor module (176) may include, for example, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0039] The interface (177) may support one or more specified protocols that can be used for the electronic device (101) to be connected directly or wirelessly to an external electronic device (e.g., electronic device (102)). According to one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, a secure digital (SD) card interface, or an audio interface.

[0040] The connection terminal (178) may include a connector through which the electronic device (101) can be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0041] The haptic module (179) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that can be perceived by the user through tactile or kinesthetic senses. According to one embodiment, the haptic module (179) may include, for example, a motor, a piezoelectric element, or an electric stimulation device.

[0042] The camera module (180) can capture still images and video. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0043] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least part of a power management integrated circuit (PMIC).

[0044] The battery (189) can supply power to at least one component of the electronic device (101). According to one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0045] The communication module (190) can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between an electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may include one or more communication processors that operate independently of the processor (120) (e.g., application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., cellular communication module, short-range wireless communication module, or GNSS (global navigation satellite system) communication module) or a wired communication module (194) (e.g., LAN (local area network) communication module, or power line communication module). The corresponding communication module among these communication modules can communicate with an external electronic device (104) through a first network (198) (e.g., a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data association)) or a second network (199) (e.g., a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can identify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) using subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module (196).

[0046] The wireless communication module (192) can support 5G networks and next-generation communication technologies following 4G networks, for example, new radio access technology. NR access technology can support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and connection of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low-latency communications (URLLC)). The wireless communication module (192) can support a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate, for example. The wireless communication module (192) can support various technologies for securing performance in the high-frequency band, such as beamforming, massive MIMO (multiple-input and multiple-output), full-dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large-scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), external electronic device (e.g., electronic device (104)), or network system (e.g., second network (199)). According to one embodiment, the wireless communication module (192) may support a Peak data rate (e.g., 20 Gbps or more) for eMBB realization, loss coverage (e.g., 164 dB or less) for mMTC realization, or U-plane latency (e.g., downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less) for URLLC realization.

[0047] An antenna module (197) can transmit a signal or power to or from an external source (e.g., an external electronic device). According to one embodiment, the antenna module (197) may include an antenna comprising a radiator made of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as a first network (198) or a second network (199), may be selected from the plurality of antennas, for example, by a communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, other components (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module (197).

[0048] According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the printed circuit board and capable of supporting a specified high frequency band (e.g., mmWave band), and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top surface or side surface) of the printed circuit board and capable of transmitting or receiving a signal of the specified high frequency band.

[0049] At least some of the above components can be connected to each other via a communication method between peripheral devices (e.g., bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)) and exchange signals (e.g., commands or data) with each other.

[0050] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) through a server (108) connected to a second network (199). Each of the external electronic devices (102, or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations performed on the electronic device (101) may be performed on one or more of the external electronic devices (102, 104, or 108). For example, if the electronic device (101) needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device (101) may request one or more external electronic devices to perform at least part of the function or service instead of performing the function or service itself or additionally. One or more external electronic devices that receive the above request may execute at least part of the requested function or service, or additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may provide the result as is or additionally processed as at least part of the response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device (101) may provide ultra-low latency services using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server using machine learning and / or neural networks. According to one embodiment, the external electronic device (104) or the server (108) may be included within a second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0051] FIG. 2 illustrates an IoT (internet of things) system (200) according to one embodiment. At least some of the components of FIG. 2 may be omitted, and the system may be implemented to include additional components not illustrated.

[0052] Referring to FIG. 2, an IoT system (200) according to one embodiment includes a plurality of electronic devices that can be connected to a network (216 or 246) (e.g., a second network (199) of FIG. 1). For example, the IoT system (200) may include at least one of a first IoT server (210), a first node (220), an A1 voice assistance server (230), a second IoT server (240), a second node (250), or devices (221, 222, 223, 224, 225, 236, 237, 251, 252, 253).

[0053] According to one embodiment, the first IoT server (210) may include at least one of a communication interface (211), a processor (212), or a storage unit (213). The second IoT server (240) may include at least one of a communication interface (241), a processor (242), or a storage unit (243). An “IoT server” in this document may remotely control and / or monitor one or more devices (e.g., devices (221, 222, 223, 224, 225, 251, 252, 253)) based on, for example, a network (e.g., network (216) or network (246)), through a relay device (e.g., first node (220) or second node (250)), or directly without a relay device. The term “device” as used herein refers to sensors, home appliances, office electronic devices, or devices for performing processes deployed (or located) within a local environment, such as a home, office, factory, building, external branch, or other types of sites, without any limitations on their type. A device that receives a control command and performs an action corresponding to the command may be referred to as a “target device.” An IoT server may also be referred to as a central server in that it selects a target device from among multiple devices and provides control commands.

[0054] According to one embodiment, the first IoT server (210) can communicate with devices (221, 222, 223) through a network (216). The network (216) may mean a network for remote communication, such as, for example, the Internet, or a computer network (e.g., LAN or WAN), or may include a cellular network.

[0055] According to one embodiment, the first IoT server (210) may be connected to a network (216) through a communication interface (211). The communication interface (211) may include a communication device (or communication module) for supporting communication of the network (216), and may be integrated into a single component (e.g., a single chip) or implemented as a plurality of separate components (e.g., multiple chips). The first IoT server (210) may communicate with devices (221, 222, 223) through a first node (220). The first node (220) may receive data from the first IoT server (210) through the network (216) and transmit the received data to at least some of the devices (221, 222, 223). Alternatively, the first node (220) may receive data from at least some of the devices (221, 222, 223) and transmit the received data to the first IoT server (210) via the network (216). The first node (220) may function as a bridge between the network (216) and the devices (221, 222, 223). Meanwhile, although FIG. 2 is illustrated as having only one first node (220), this is merely illustrative and there is no limit to the number.

[0056] In this document, the “node” may be an edge computing system or a hub device. According to one embodiment, the first node (220) supports wired and / or wireless communication of the network (216) and may also support wired and / or wireless communication with devices (221, 222, 223). For example, the first node (220) may be connected to the devices (221, 222, 223) via a short-range communication network such as at least one of Bluetooth, WiFi, WiFi direct, Z-wave, Zig-Bee, INSETEON, X10, or IrDA (infrared data association), but there is no limitation on the type of communication. The first node (220) may be deployed (or located) in an environment such as, for example, a home, office, factory, building, external branch, or other types of sites. Accordingly, the devices (221, 222, 223) may be monitored and / or controlled by a service provided by the first IoT server (210), and the devices (221, 222, 223) may not be required to have the capability of full network communication (e.g., Internet communication) for direct connection to the first IoT server (210). The devices (221, 222, 223) are illustrated as being implemented as electronic devices within a home environment, such as light switches, proximity sensors, and temperature sensors, but this is exemplary and not limited thereto.

[0057] According to one embodiment, the first IoT server (210) may support direct communication with devices (224, 225). Here, “direct communication” refers to communication without passing through a relay device, such as, for example, the first node (220), and may mean, for example, communication through a cellular communication network and / or a network.

[0058] According to one embodiment, the first IoT server (210) may transmit a control command to at least some of the devices (221, 222, 223, 224, 225). Here, “control command” may mean data that causes a controllable device to perform a specific action, and the specific action is an action performed by the device, which may include outputting information, sensing information, reporting information, and managing information (e.g., deletion, or creation), and there is no limitation on the type thereof. For example, the processor (212) may obtain information (or a request) for generating a control command from an external electronic device (e.g., A1 agent server (230), second IoT server (240), external system (260), or at least some of the devices (221, 222, 223, 224, 225), and generate a control command based on the obtained information. Alternatively, the processor (212) may generate a control command based on the fact that the monitoring results of at least some of the devices (221, 222, 223, 224, 225) satisfy a specified condition. Alternatively, the processor (212) may receive a control command from an external electronic device. The processor (212) may control the communication interface (211) to transmit the control command to a target device.

[0059] According to one embodiment, the processor (212), processor (232), or processor (242) may be implemented as a combination of one or more general-purpose processors such as a CPU (central processing unit), DSP (digital signal processor), AP (application processor), CP (communication processor), graphics-dedicated processors such as a GPU (graphical processing unit) and VPU (vision processing unit), or artificial intelligence-dedicated processors such as an NPU (neural processing unit). The processing units described above are merely exemplary, and those skilled in the art will understand that the processor (212) is not limited to any means of computation capable of executing instructions stored in memory (213), for example, and outputting the result of the execution.

[0060] According to one embodiment, the processor (212) may configure a web-based interface based on an API (214) or expose resources managed by the first IoT server (210) to the outside. The web-based interface may support, for example, communication between the first IoT server (210) and an external web service. The processor (212) may allow, for example, an external system (260) to control and / or access the devices (221, 222, 223). The external system (260) may, for example, be an independent system that is not associated with or part of the system (200). The external system (260) may, for example, be an external server or a website. However, security is required for access to the devices (221, 222, 223) or resources of the first IoT server (210) from the external system (260). According to one embodiment, the processor (212) may expose an API endpoint (e.g., a URL (universal resource locator)) based on the API (214) to the outside of the automation application. As described above, the first IoT server (210) may transmit control commands to a target device among the devices (221, 222, 223). Meanwhile, the description of the communication interface (241), processor (242), API (244) of the storage unit (243), and database (245) of the second IoT server (240) may be substantially the same as the description of the communication interface (211), processor (212), API (214) of the storage unit (213), and database (215) of the first IoT server (210). Additionally, the description of the second node (250) may be substantially the same as the description of the first node (220). The second IoT server (240) can transmit control commands to a target device among the devices (251, 252, 253).The first IoT server (210) and the second IoT server (240) may be operated by the same service provider in one embodiment, but may be operated by different service providers in another embodiment.

[0061] According to one embodiment, the A1 agent server (230) can transmit and receive data with the first IoT server (210) through a network (216). The A1 agent server (230) according to one embodiment may include at least one of a communication interface (231), a processor (232), or a storage unit (233). The communication interface (231) can communicate with a smartphone (236) or an AI speaker (237) through a network (not shown) and / or a cellular network (not shown). The smartphone (236) (e.g., the electronic device (101) of FIG. 1) or the AI ​​speaker (237) (e.g., the electronic device (102) of FIG. 1) may include a microphone and can acquire a user voice, convert it into a voice signal, and transmit the voice signal to the A1 agent server (230). The processor (232) can receive a voice signal from a smartphone (236) or an AI speaker (237) through a communication interface (231). The processor (232) can process the received voice signal based on a stored model (e.g., LLM) (234). The processor (232) can generate (or verify) a control command using the processing result based on information stored in a database (235).According to one embodiment, the storage unit (213, 233, 243) may include at least one type of non-transitory storage medium among a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD (secure digital) or XD (extreme digital) memory), RAM (random access memory), SRAM (static random access memory), ROM (read-only memory), EEPROM (electrically erasable programmable read-only memory), PROM (programmable read-only memory), magnetic memory, a magnetic disk, and an optical disk, and there is no limitation on the type.

[0062] FIG. 3 illustrates an AI agent service system (hereinafter simply the system) (300) for responding more accurately to a user's speech according to one embodiment. Referring to FIG. 3, the system (300) may include a listening device (or a first electronic device) (310), a master device (or a second electronic device) (320), a group of IoT devices (330), and an AI agent server (340). The devices (310, 320, 340, 350) of FIG. 3 may exchange data with each other through a network (350) (e.g., the network (198 and / or 199) of FIG. 1 or the network (216 and / or 246) of FIG. 2).

[0063] The listening device (310) can listen to the user's speech. The master device (320) can perform actions to control and manage the IoT device group (330) overall to respond more accurately to the user's speech. According to one embodiment, an AI agent server (340) may perform the control and management actions on behalf of the master device (320). The listening device (310) corresponds to the electronic device (102) in FIG. 1 or the AI ​​speaker (237) in FIG. 2, and the master device (320) may correspond to the electronic device (101) in FIG. 1. The master device (320) may correspond to a portable electronic device (e.g., a smartphone (236) in FIG. 2). The master device (320) may be a wearable device. For example, a head-mounted display (HMD) device that supports augmented reality (AR) and / or virtual reality (VR) may be implemented as the master device (320). The master device (320) may be an IoT device within the IoT device group (330). For example, a home appliance such as a TV or refrigerator, or a hub device connecting the network (350) and the IoT device (e.g., the first node (220) or the second node (250) in FIG. 2) may be implemented as the master device (320). According to one embodiment, the listening device (310) and the master device (320) may be electronic devices integrated into a single electronic device (e.g., the electronic device (101) in FIG. 1, a portable electronic device, a wearable electronic device, an IoT device, or a hub device). For example, the operation of controlling and managing the listening and the IoT device group (330) overall may be implemented in a single electronic device.

[0064] The listening device (310) may include a first microphone (311), a first communication circuit (312), a first memory (318), and a first processor (319). The first communication circuit (312), the first memory (318), and the first processor (319) may each perform the same function by being implemented substantially identically to the communication module (190), memory (130), and processor (120) in the electronic device (101) of FIG. 1. The first microphone (311) may be a component of the input module (150) in FIG. 1. The microphone (311) may be configured to receive external sound and convert the sound into an analog electrical signal (audio signal). The audio signal may be converted into a digital electrical signal (audio data) through an analog-to-digital converter (ADC) of the listening device (310) and output to other components of the listening device (310) (e.g., a first processor (319) and / or a first communication circuit (312)). Instructions may be stored in the first memory (318). A portion of the instructions may be stored in the first memory (318) and another portion may be stored in the internal memory of the first processor (319). When the instructions are executed by the first processor (319), they may cause the listening device (310) to perform a given operation.

[0065] The master device (320) may include a second communication circuit (322), a second memory (328), and a second processor (329). The second communication circuit (322), the second memory (328), and the second processor (329) may each perform the same function by being implemented substantially identically to the communication module (190), memory (130), and processor (120) in the electronic device (101) of FIG. 1. Instructions may be stored in the second memory (328). A portion of the instructions may be stored in the second memory (328) and another portion may be stored in the internal memory of the second processor (329). When the instructions are executed by the second processor (329), they may cause the master device (320) to perform a given operation. According to one embodiment, the master device (320) may directly listen to the user's speech. Accordingly, the master device (320) may further include a second microphone (312).

[0066] The IoT device group (330) may include IoT devices (e.g., TV, washing machine, light switch, fan, air conditioner) located in a designated place (e.g., home). For example, the IoT device group (330) may include devices (221, 222, 223) connected to the first IoT server (210) and the first node (220) in FIG. 2. Alternatively, the IoT device group (330) may include devices (251, 252, 253) connected to the second IoT server (240) and the second node (250) in FIG. 2. Communication for data transmission and reception or remote control may be performed between IoT devices within the IoT device group (330) through the network (350) or an access point (AP) located at the place, or by using peer-to-peer (P2P) communication technology (e.g., Wi-Fi Direct or Bluetooth).

[0067] An AI agent server (340) (e.g., an AI agent server (230)) can generate a control command to control the operation of at least one IoT device belonging to an IoT device group (330) based on a user utterance obtained from a listening device (310) and transmit it to the IoT device group (330).

[0068] FIG. 4 is a diagram illustrating an operation for updating a prompt to be input into a small LLM so as to respond more accurately to a user's utterance in the system (300) of FIG. 3 according to one embodiment. Referring to FIG. 4, the system (300) may include a listening device (310), a plurality of control modules (421, 422, 423), a plurality of IoT devices (431, 432, 433), and a master device (320).

[0069] The listening device (310) may include a broadcasting module (410). The broadcasting module (410) may be composed of instructions and stored in the internal memory of the first memory (318) and / or the first processor (319). The broadcasting module (410) may acquire user speech from the first microphone (311) and transmit the user speech to a plurality of control modules (421, 422, 423) and a master device (320).

[0070] According to one embodiment, the broadcasting module (410) may transmit metadata of a user utterance along with the user utterance. The metadata may include, for example, information indicating the location of the IoT device group to which the listening device (310) belongs (or the user who made the utterance), information indicating the specific locale where the user utterance was acquired at that location (e.g., the location of the listening device (310)), information indicating the type of the listening device (310), or information indicating the timestamp at which the user utterance was acquired.

[0071] According to one embodiment, the listening device (310) may respond only to the utterance of a user configured to receive an AI agent service. For example, feature data of a user's voice registered in the system (300) may be stored in the first memory (318) of the listening device (310). The listening device (310) may determine whether the speaker is a registered user by measuring the similarity between the acquired utterance and the feature data of the user's voice stored in the first memory (318). The listening device (310) may use an artificial intelligence model for similarity measurement. For example, the listening device (310) may input a prompt (input value) containing the speaker's voice data and the feature data of the registered user's voice into the artificial intelligence model and obtain a result value (e.g., information indicating similarity and / or whether the two voices match) from the artificial intelligence model. The broadcasting module (410) may transmit the user's utterance based on the determination that the speaker is a registered user.

[0072] A plurality of control modules (421, 422, 423) may correspond one-to-one with a plurality of IoT devices (431, 432, 433) in an IoT device group (330). For example, a first control module (421) can control the operation of a first IoT device (431) in an IoT device group (330). A second control module (422) can control the operation of a second IoT device (432) in an IoT device group (330). A third control module (423) can control the operation of a third IoT device (433) in an IoT device group (330). The number of IoT devices and corresponding control modules illustrated in FIG. 4 is exemplified as three each, but this example is not to limit the embodiments of the present disclosure. For example, a number of IoT devices (e.g., light switch, fan, washing machine, robot vacuum cleaner, TV, air conditioner) can be members of an IoT device group (330) in a list of devices stored in a master device (320) (e.g., stored in a second memory (328)) and managed through one or more designated user accounts.

[0073] Among the plurality of control modules (421, 422, 423), the first control module (421) may include a prompt generation module (441) and a small LLM (443). The prompt generation module (441) can generate a prompt (input value) to be input into the small LLM (443) using a user utterance received from the listening device (310) and a prompt template (445).

[0074] According to one embodiment, the prompt generation module (441) can complete the prompt by adding (or injecting) user utterance into a pre-prepared prompt template (445).

[0075] According to one embodiment, the prompt generation module (441) can complete the prompt by adding the user's context information to the prompt template (445) along with the user's utterance. Here, the user context information may include metadata received from the listening device (310) along with the user's utterance.

[0076] According to one embodiment, the prompt generation module (441) can complete the prompt by adding the user's context information and the context information of the corresponding IoT device to the prompt template (445) along with the user's utterance. For example, the prompt generation module (441) can receive device context information of the first IoT device (431) from the first IoT device (431). For example, the device context information may include a description of the IoT device, information indicating the type of the IoT device, information indicating the location where the IoT device is located, or information indicating the specific locale where the IoT device is located at that location.

[0077] In this document, a 'prompt template' may include instructions (or goals) indicating a task to be performed, which an AI model (e.g., LLM) utilizes input values ​​added to the template (e.g., content of user utterance, user context information, context information of the corresponding IoT device). Based on the instructions, the AI ​​model can analyze the user utterance to generate and output a response (e.g., a control command indicating what the corresponding IoT device should do).

[0078] A small LLM (443) can generate a control command for controlling a first IoT device (431) using a prompt input from a prompt generation module (441). The control command is transmitted to the first IoT device (431), so that the first IoT device (431) can perform the corresponding function. For example, if the first IoT device (431) is an air conditioner, a control command “Turn-on” can be transmitted from the small LLM (443) to the first IoT device (431) in response to a user’s utterance “It is too hot.” ​​The first IoT device (431), i.e., the air conditioner, can turn on the power and perform a cooling function in response to the received control command.

[0079] The small LLM (443) can also transmit control commands generated based on the prompt to the master device (320).

[0080] The second control module (422) may also be composed of the same prompt generation module and small LLM as the prompt generation module (441) and small LLM (443). The third control module (423) may also be composed of the same prompt generation module and small LLM as the prompt generation module (441) and small LLM (443).

[0081] A plurality of control modules (421, 422, 423) may each be configured in the corresponding IoT device or in the AI ​​agent server (340). For example, in the first control module (421), the prompt generation module (441) may be included in the first IoT device (431) and the small LLM (443) may be included in the AI ​​agent server (340). The prompt generation module (441) may be composed of instructions and stored in the memory of the first IoT device (431). The instructions may cause the first IoT device (431) to generate a prompt when executed by the processor of the first IoT device (431). As another example, both the prompt generation module (441) and the small LLM (443) may be included in the first IoT device (431).

[0082] The master device (320) may include a first prompt generation module (451), a large LLM (453), and a prompt update module (455). The first prompt generation module (451) and the prompt update module (455) may be composed of instructions and stored in the internal memory of the second memory (328) and / or the second processor (329). The large LLM (453) may be stored in the second memory (328) and executed by the second processor (329). According to one embodiment, the large LLM (453) may be included in the AI ​​agent server (340). Accordingly, the master device (320) may transmit the prompt generated by the first prompt generation module (451) to the AI ​​agent server (340) via the second communication circuit (322). The master device (320) can receive results (e.g., updates) generated by the AI ​​agent server (340) from the AI ​​agent server (340) through the second communication circuit (322).

[0083] According to one embodiment, the first prompt generation module (451) can receive a user utterance from a listening device (310) and receive output values ​​(control commands) for each of a number of small LLMs from a number of control modules (421, 422, 423). The first prompt generation module (451) can complete the prompt by adding the user utterance and the number of output values ​​(control commands) to the first prompt template (457).

[0084] According to one embodiment, the first prompt generation module (451) can receive user utterance and user context information from the listening device (310) and receive output values ​​(control commands) of each of the multiple small LLMs from the multiple control modules (421, 422, 423). The first prompt generation module (451) can complete the prompt by adding the user utterance, user context information, and multiple output values ​​(control commands) to the first prompt template (457).

[0085] According to one embodiment, the first prompt generation module (451) can receive user utterance and user context information from the listening device (310). The first prompt generation module (451) can receive context information of the corresponding IoT device along with a plurality of output values ​​(control commands) from a plurality of control modules (421, 422, 423). The first prompt generation module (451) can complete the prompt by adding the user utterance, user context information, a plurality of output values ​​(control commands), and context information of the plurality of IoT devices to the first prompt template (457).

[0086] A large-scale LLM (453) can determine an update target among multiple IoT devices (431, 432, 433) using a prompt input from a first prompt generation module (451) and generate update content to be delivered to the control module of the update target. A prompt update module (455) can deliver the update content to the control module of the IoT device determined as the update target. The control module can generate a prompt based on the update content. For example, the control module can add update content to a prompt template or modify existing content (e.g., instructions) in a prompt template based on the update content.

[0087] FIGS. 5a, 5b, 5c, and 5d are drawings for illustrating an example of an AI agent service according to one embodiment.

[0088] Referring to FIG. 5a, a user utterance (591) saying “it is too hot,” user context information (592), and device context information (581) of the air conditioner (501) can be input into the prompt generation module (511) of the air conditioner (501). The prompt generation module (511) can generate a prompt (531) by injecting the input content into a prompt template (521). As illustrated, the prompt template (521) may include a number of major items (SYSTEM, USER CONTEXT, DEVICE CONTEXT, FUNTION LIST, SPECIAL ORDER). The prompt template (521) may include one or more minor items belonging to a major item. For example, the prompt template (521) may include “timestamp,” “location,” “locale,” “type,” and “request” as minor items belonging to [USER CONTEXT]. The prompt template (521) may include “description of device”, “device type”, “location”, and “locale” as sub-items belonging to [DEVICE CONTEXT]. The prompt template (521) may include information indicating functions that can be performed by the air conditioner (501). For example, “On”, “Off”, “Raise temperature”, “Down temperature”, and “No wind” may be recorded as functions of the air conditioner (501) in [FUCTION LIST]. The prompt template (521) may include updates provided by the master device (320). For example, updates may be recorded in the [SPECIAL ORDER] item. According to the example in FIG. 5a, it is currently not recorded. The prompt template (521) may include instructions indicating tasks that the small LLM (541) must perform. For example, instruction a may be recorded in the [SYSTEM] item. An example of instruction a is as follows.

[0089] Instruction a: You are an air conditioner. Listen to user requests and provide appropriate responses, or invoke functions such as playing music. The list of available functions is in the [FUNCTION LIST], which you can use to execute. When analyzing requests, you can receive user requests via [USER CONTEXT] and know your current status via [DEVICE CONTEXT]. Before performing a function, you must refer to the [SPECIAL ORDER] to ensure that it does not violate its contents. The [SPECIAL ORDER] consists of the priority items requested by the user.

[0090] Referring again to FIG. 5a, the prompt generation module (511) can input the prompt (531) into a small LLM (541). The small LLM (541) can output a control command a “Turn on” to the air conditioner (501) and the master device (320) as a result obtained by processing the prompt (531).

[0091] Referring to FIG. 5b, a user utterance (591) saying “it is too hot,” user context information (592), and device context information (582) of the fan (502) can be input into the prompt generation module (513) of the fan (502). The prompt generation module (513) can generate a prompt (533) by injecting the input content into a prompt template (523). The prompt template (523) may be written in the same format as the prompt template (521) of FIG. 5a described above. In the prompt template (523), instructions b related to the operation of the fan (502) may be recorded in the [SYSTEM] item. An example of instructions b is as follows.

[0092] Instruction b: You are a ventilation fan. You can respond to user requests. The list of available functions is in the [FUNCTION LIST], and you can execute the corresponding functions using it. When analyzing requests, you can receive user requests via [USER CONTEXT] and check your current status via [DEVICE CONTEXT]. Before performing a function, you must refer to the [SPECIAL ORDER] to ensure that it does not violate its contents. The [SPECIAL ORDER] consists of the priority items requested by the user.

[0093] Referring again to FIG. 5b, the prompt generation module (513) can input the prompt (533) into the small LLM (543). The small LLM (543) can output the control command b “Turn on” to the fan (502) and the master device (320) as a result obtained by processing the prompt (533).

[0094] Referring to FIG. 5c, a user utterance (591) saying “it is too hot,” user context information (592), and control commands (a, b, …) and device context information (581, 582, …) for multiple IoT devices can be input into a first prompt generation module (451) of a master device (320). The first prompt generation module (451) can generate a prompt (535) by adding (or injecting) the input content into a first prompt template (457). The first prompt template (457) may include multiple major items (SYSTEM, USER REQUEST, DEVICE ACTION #1, DEVICE ACTION #2, …, DEVICE ACTION #n) as illustrated. [USER REQUEST] may include “timestamp,” “location,” “locale,” and “request” as corresponding minor items. [DEVICE ACTION] may include “description of device”, “device type”, “location”, “locale”, and “action”. The first prompt template (457) may include instructions indicating the action that the large LLM (453) must perform. For example, instruction c may be recorded in the [SYSTEM] entry. An example of instruction c is as follows.

[0095] Instruction c: You are a Smart Home Manager. Your role is to receive and analyze the execution of devices within the space you manage, and if a device attempts to perform an incorrect action, to reject the execution and insert a prompt to correct it. You receive [USER REQUEST], which is the request received by the device, and [DEVICE ACTION], which is the action performed by the device, and make judgments based on them. Your goal is to analyze the behavior of devices executed around the same time and correct them.

[0096] Referring again to FIG. 5c, the first prompt generation module (451) can input the prompt (535) into the large-scale LLM (453). The large-scale LLM (453) can output information regarding the update target and the update content as a result obtained by processing the prompt (535) to the prompt update module (455). The prompt update module (455) can transmit the update content to the prompt generation module of the IoT device determined as the update target. For example, as the update target is determined to be the ventilation fan (502), the update content (565) stating “does not react to ignition related to temperature” can be transmitted to the prompt generation module (513) of the ventilation fan (502). According to the embodiment of FIG. 5c, the large-scale LLM (453) can correct errors in the small-scale LLM.

[0097] Referring to FIG. 5d, a user utterance (593) saying “Ah, it’s hot,” user context information (594), and device context information (583) of the fan (502) can be input into the prompt generation module (513) of the fan (502). The prompt generation module (513) can generate a prompt (573) by adding (or injecting) the input content into a prompt template (523) updated by the master device (320). For example, in the prompt template (523), an instruction (571) stating “do not respond to utterances related to temperature” can be recorded in [SPECIAL ORDER]. The prompt generation module (513) can input the prompt (573) into a small LLM (543). The small LLM (543) may not respond to the user utterance (593) based on the instruction (571). Accordingly, control commands related to the ventilation fan (502) may not be output to the ventilation fan (502) and the master device (320).

[0098] FIG. 6 is a diagram illustrating an operation to update a prompt to be input into a small LLM so as to respond more accurately to a user's utterance in the master device (320) of FIG. 3 according to one embodiment.

[0099] Referring to FIG. 6, the master device (320) may include a large LLM (453), a prompt update module (455), and a second prompt generation module (610). The master device (320) may acquire user speech through a second microphone (312). Alternatively, the master device (320) may obtain user utterance from an external electronic device (e.g., a listening device (310)) through the second communication circuit (322). The second prompt generation module (610) may be composed of instructions and stored in the second memory (328) and / or the internal memory of the second processor (329). The IoT device list (620) may be stored in the second memory (328) and / or the internal memory of the second processor (329). The IoT device list (620) may include device context information for each IoT device. For example, the device context information may include the name of the IoT device, information indicating the type of the IoT device, information indicating the location where the IoT device is located, or information indicating the specific locale where the IoT device is located at that location.

[0100] The second prompt generation module (610) can obtain a user utterance and an IoT device list (620) and use the user utterance and the IoT device list (620) to generate a prompt (input value) to be input into a large-scale LLM (453). According to one embodiment, the second prompt generation module (610) can complete the prompt by adding (or injecting) the user utterance and the IoT device list (620) into a second prompt template (630). The large-scale LLM (453) can output information regarding the update target and content to the prompt update module (455) as a result obtained by processing the prompt generated by the second prompt generation module (610). The prompt update module (455) can transmit the update content to the control module of the IoT device determined as the update target.

[0101] FIG. 7 is a diagram illustrating an example of an AI agent service according to one embodiment.

[0102] Referring to FIGS. 6 and 7, a user utterance (791) “make it dark when watching a movie,” user context information (792), and a list of IoT devices (620) can be input into a second prompt generation module (610). The second prompt generation module (610) can generate a prompt (710) by injecting the input content into a second prompt template (630). The second prompt template (630) may include a number of major items (SYSTEM, USER REQUEST, DEVICE #1, DEVICE #2, …, DEVICE #n) as illustrated. [USER REQUEST] may include “timestamp,” “location,” “locale,” and “request” as corresponding minor items. [DEVICE] may include “device name,” “device type,” “location,” and “locale.” The third prompt template (610) may contain instructions indicating the work that the large LLM (453) must do. For example, instruction d may be written in the [SYSTEM] entry. An example of instruction d is as follows.

[0103] Instruction d: You are a Smart Home Manager. You can update the prompt templates of IoT devices to process user requests by utilizing user requests and device information from the spaces you manage. You can update the [SPECIAL ORDER] of IoT devices. The updated content is used as reference material when the corresponding IoT device executes specific functions.

[0104] The second prompt generation module (610) can input the prompt (710) into a large-scale LLM (453). The large-scale LLM (453) can output information regarding the update target and the update content as a result obtained by processing the prompt (710) to the prompt update module (455). The prompt update module (455) can transmit the update content to the prompt generation module of the IoT device determined as the update target. For example, as the update target is determined to be a ceiling light (e.g., LED light), the update content (720) “set brightness to dark when watching video” can be transmitted to the ceiling light prompt generation module (730). The ceiling light prompt generation module (730) can update the prompt template (740) by recording the update content (720) in [SPECIAL ORDER] in the corresponding prompt template (740). An example of instruction e in the ceiling light prompt template (740) is as follows.

[0105] Instruction e: You are a ceiling light. You can listen to user requests and provide appropriate responses. The list of available functions is in the [FUNCTION LIST], and you can execute the corresponding functions using it. When analyzing requests, you can receive user requests through [USER CONTEXT] and know your current status through [DEVICE CONTEXT]. Before performing a function, you must refer to the [SPECIAL ORDER] to ensure that it does not violate its contents. The [SPECIAL ORDER] is the priority requested by the user.

[0106] FIG. 8 is a flowchart illustrating the operation of an electronic device (e.g., master device (320) of FIG. 3) for updating a prompt according to one embodiment. According to one embodiment, a processor (e.g., a second processor (329)) in the electronic device may be configured to perform the operation of FIG. 8. According to one embodiment, instructions stored in memory (e.g., second memory (328) and / or internal memory of the second processor (329)) may cause the electronic device to perform the operation of FIG. 8 when executed by the processor. FIG. 9 illustrates an example of a message provided to a user upon updating a prompt.

[0107] In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

[0108] In operation 810, the electronic device can obtain information regarding at least one control command that controls at least one IoT device in a group of IoT devices (e.g., the group of IoT devices (330) of FIG. 3) and the user's utterance.

[0109] User utterance may be acquired through a microphone of an electronic device or acquired from an external electronic device (e.g., a listening device (310)) and received by the electronic device through a communication circuit. According to one embodiment, the electronic device may acquire user context information along with the utterance. User context information, in other words as metadata of the utterance, may include information indicating at least one of, for example, the timestamp at which the user uttered, the location at which the utterance was uttered, the locale at the location, or the type of device that acquired the utterance (e.g., an AI speaker or a smartphone).

[0110] An IoT device can perform functions according to a control command. According to one embodiment, if the IoT device is equipped with a small LLM that generates a control command based on user utterance, an electronic device may receive information regarding the control command from the IoT device that performed the function according to the control command. According to one embodiment, the small LLM may be located in an external electronic device (e.g., an AI agent server (340)) from the perspective of the IoT device. Accordingly, the electronic device may receive information regarding the control command from a device equipped with the small LLM that generates the control command, rather than from the IoT device. According to one embodiment, even if the small LLM is located in an external electronic device from the perspective of the IoT device, the IoT device may receive the control command from the external electronic device and transmit it to the electronic device.

[0111] According to one embodiment, an electronic device may receive device context information from an IoT device that has performed a function in accordance with a control command. For example, the electronic device may receive device context information from the IoT device along with a control command. The device context information, expressed differently as state information of the device, may include information indicating at least one of, for example, the current operating state of the device (description of device), the type of the device, the location where the device is located, and a specific locale at the location.

[0112] In operation 820, the electronic device may generate a first prompt to be input into a first LLM (e.g., a large-scale LLM (453)) based on information regarding a user's utterance and at least one control command. According to one embodiment, the electronic device may have the first prompt generation module (451) described above. The electronic device may generate a first prompt (input value) by using the first prompt generation module (451) to inject the user utterance, user context information, control command, and device context information of the IoT device that performed the function according to the control command into a first prompt template (457).

[0113] In operation 830, the electronic device can identify the update target and the associated update content in the output value of the first LLM. Here, the output value is the value output by the first LLM as a result of the first prompt being input into the first LLM. The update target is a device selected by the first LLM among the at least one external electronic device. The update content is part of the second prompt to be input into the second LLM (e.g., small LLM (543)) associated with the update target. The update content identifiable in the output value may include instructions indicating the action (or rule or goal) that the second LLM must perform based on the second prompt. For example, the update content may include instructions (e.g., update content (565) in FIG. 5c) that can be added to the prompt template (e.g., prompt template (523) in FIG. 5b).

[0114] According to one embodiment, the electronic device may display a message on the display of the electronic device to allow the user to recognize the response of the IoT device to their utterance. For example, referring to FIG. 9, an update (910) indicating that the fan will no longer respond to a temperature-related utterance, such as the user’s “too hot,” may be displayed.

[0115] According to one embodiment, the electronic device may output a voice corresponding to the update content through the speaker of the electronic device. Alternatively, the electronic device may control the sound output device through the communication circuit of the electronic device so that the voice is output to an external sound output device (e.g., an AI speaker in the IoT device group (330)).

[0116] In operation 840, the electronic device may transmit the update content to an external electronic device configured to generate a second prompt. Here, the external electronic device may be a device having a prompt generation module (e.g., the prompt generation module (513) of FIG. 5b). This device may be an IoT device determined as the update target. Alternatively, the external electronic device may be an external server (e.g., the AI ​​agent server (340) of FIG. 3).

[0117] FIG. 10 is a flowchart illustrating the operation of an electronic device (e.g., master device (320) of FIG. 3) for updating a prompt according to one embodiment. According to one embodiment, a processor (e.g., a second processor (329)) in the electronic device may be configured to perform the operation of FIG. 10. According to one embodiment, instructions stored in memory (e.g., second memory (328) and / or internal memory of the second processor (329)) may cause the electronic device to perform the operation of FIG. 10 when executed by the processor. FIG. 11 illustrates an example of a message provided to a user upon updating a prompt.

[0118] In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

[0119] In operation 1010, the electronic device may acquire a user's utterance. The user's utterance may be acquired through a microphone of the electronic device or acquired from an external electronic device (e.g., a listening device (310)) and received by the electronic device through a communication circuit. According to one embodiment, the electronic device may acquire user context information along with the utterance. User context information, in other words, as metadata of the utterance, may include information indicating at least one of, for example, the timestamp at which the user uttered, the location at which the utterance was uttered, the locale at the location, or the type of device that acquired the utterance (e.g., an AI speaker or a smartphone).

[0120] In 1020, the electronic device can generate a first prompt to be input into a first LLM (e.g., a large-scale LLM (453)) based on the user's utterance and a list of devices registered to the electronic device (e.g., the list of IoT devices (620) in FIG. 6). For example, the list of devices may be stored in the memory of the electronic device (e.g., a second memory (328)). According to one embodiment, the electronic device may have the second prompt generation module (610) described above. The electronic device can generate a first prompt (input value) by using the second prompt generation module (610) to inject the user utterance and the list of devices into a second prompt template (630).

[0121] In operation 1030, the electronic device can check the update content of the second prompt to be entered into the second LLM (e.g., small LLM (543)) in the device list as a result of inputting the first prompt into the first LLM output value. The update content that can be checked in the output value may include instructions indicating the action (or goal) that the second LLM must perform based on the second prompt. For example, the update content may include instructions (e.g., update content (720) of FIG. 7) that can be added to the prompt template (e.g., prompt template (740) of FIG. 7).

[0122] According to one embodiment, the electronic device may display a message on the display of the electronic device to allow the user to recognize the response of the IoT device to their utterance. For example, referring to FIG. 11, an update (1110) indicating that the living room light will respond to a brightness-related utterance, such as the user’s “I would like the surroundings to be dark when watching a movie,” may be displayed.

[0123] According to one embodiment, the electronic device may output a voice corresponding to the update content through the speaker of the electronic device. Alternatively, the electronic device may control the sound output device through the communication circuit of the electronic device so that the voice is output to an external sound output device (e.g., an AI speaker in the IoT device group (330)).

[0124] In operation 1140, the electronic device may transmit the update content to an external electronic device configured to generate a second prompt. Here, the external electronic device may be a device having a prompt generation module (e.g., a prompt generation module having the prompt template (740) of FIG. 7). This device may be an IoT device determined as the update target. Alternatively, the external electronic device may be an external server (e.g., an AI agent server (340) of FIG. 3).

[0125] According to various embodiments of the present disclosure described above, an electronic device can identify the user's unclear intent and control the IoT device. By separating the roles of a low-cost small LLM and a relatively high-cost large LLM, the response speed to user utterances can be fast and the cost for processing utterances can be reduced. Furthermore, the low-cost small LLM can be installed on the IoT device rather than on a server. Accordingly, the small LLM can respond quickly to the user's utterance on the IoT device. An incorrect response from the small LLM can be corrected by the large LLM (453). The IoT device can respond in a customized manner to the requests of individual users. For example, if a user states the AI ​​agent service they want (e.g., “Darken the screen when watching movies”), the electronic device can reflect instructions for providing the service the user wants in the prompt of the IoT device.

[0126] According to one embodiment, an electronic device (e.g., master device (320)) is provided, comprising: a microphone; a communication circuit for communicating with an external electronic device; a memory for storing instructions; and at least one processor. When the above command is executed individually or collectively by the at least one processor, the electronic device acquires a user’s utterance (e.g., "It is too hot") through the microphone or the communication circuit, receives information regarding at least one control command from the at least one external electronic device through the communication circuit, and based on the received information and the utterance, generates a first prompt to be input into a first LLM (large language model) (e.g., large LLM), checks an update target (e.g., fan) and a second prompt to be input into a second LLM (e.g., small LLM) related to the update target from the output value output from the first LLM as a result of inputting the first prompt into the at least one external electronic device (e.g., SPECIAL ORDER), and transmits the update content to an external electronic device configured to generate the second prompt through the communication circuit.

[0127] When the above command is executed individually or collectively by the at least one processor, the electronic device may generate the first prompt by injecting information regarding the utterance and the at least one control command into a pre-prepared prompt template. The prompt template may include instructions indicating what the first LLM is to do.

[0128] When the above command is executed individually or collectively by the at least one processor, the electronic device may generate the first prompt by acquiring user context information related to the utterance through the microphone or the communication circuit, and injecting information regarding the utterance, the at least one control command, and the user context information into the prompt template. The user context information may include information indicating at least one of the location where the at least one external electronic device is located, the region where the device that acquired the utterance is located at said location, the type of the device that acquired the utterance, and the time at which the utterance was acquired.

[0129] When the above instruction is executed individually or collectively by the at least one processor, the electronic device may receive device context information for each of the at least one external electronic device from the at least one external electronic device through the communication circuit, and generate the first prompt in the prompt template, the utterance, the user context information, information regarding the at least one control command, and the device context information for each of the at least one external electronic device. The device context information may include information indicating at least one of a description of the device corresponding to the source of the device context information, the type of the device, the location where the device is located, and a specific region where the device is located.

[0130] The above memory may include the first LLM.

[0131] When the above instruction is executed individually or collectively by the at least one processor, the electronic device may transmit the first prompt to an external electronic device having the first LLM through the communication circuit and receive the output value through the communication circuit.

[0132] According to one embodiment, a method for operating an electronic device (e.g., a master device (320)) is provided. The method may include: acquiring a user’s utterance through a microphone of the electronic device or a communication circuit of the electronic device; receiving information regarding at least one control command to control at least one external electronic device from the at least one external electronic device through the communication circuit; generating a first prompt to be input into a first LLM (large language model) based on the received information and the utterance; checking the update content of a second prompt to be inserted into an update target and a second LLM associated with the update target from an output value output from the first LLM as a result of inputting the first prompt into the first LLM from the at least one external electronic device; and transmitting the update content to an external electronic device configured to generate the second prompt through the communication circuit.

[0133] The operation of generating the first prompt may include generating the first prompt by injecting information regarding the utterance and the at least one control command into a pre-prepared prompt template. The prompt template may include instructions indicating what the first LLM is to do.

[0134] The above method may further include an operation of acquiring user context information related to the utterance through the microphone or the communication circuit. The operation of generating the first prompt may include an operation of generating the first prompt by injecting information regarding the utterance, the at least one control command, and the user context information into the prompt template. The user context information may include information indicating at least one of the location where the at least one external electronic device is located, the region where the device that acquired the utterance is located at said location, the type of the device that acquired the utterance, and the time at which the utterance was acquired.

[0135] The above method may further include the operation of receiving device context information for each of the at least one external electronic device from the at least one external electronic device through the communication circuit. The operation of generating the first prompt may include the operation of generating the first prompt in the prompt template, the information regarding the utterance, the at least one control command, the user context information, and the device context information for each of the at least one external electronic device. The device context information may include information indicating at least one of a description of the device corresponding to the source of the device context information, the type of the device, the location where the device is located, and a specific region where the device is located.

[0136] According to one embodiment, an electronic device (e.g., master device (320)) is provided, comprising: a microphone; a communication circuit for communicating with an external electronic device; a memory for storing instructions; and at least one processor. When the instructions are executed individually or collectively by the at least one processor, the electronic device may acquire a user’s utterance (e.g., "make it dark when watching a movie") through the microphone or the communication circuit, generate a first prompt to be input into a first LLM (e.g., large language model) based on the utterance and a list of devices registered with the electronic device, and, as a result of inputting the first prompt into the first LLM, check the update content of a second prompt to be input into a second LLM (e.g., small language model) related to an update target (e.g., light) in the list of devices and the output value output from the first LLM, and transmit the update content through the communication circuit to an external electronic device configured to generate the second prompt.

[0137] When the above command is executed individually or collectively by the at least one processor, the electronic device may generate the first prompt by injecting the utterance and the device list into a pre-prepared prompt template. The prompt template may include instructions indicating what the first LLM should do.

[0138] When the above command is executed individually or collectively by the at least one processor, the electronic device may generate the first prompt by acquiring user context information related to the utterance through the microphone or the communication circuit and injecting the utterance, the device list, and the user context information into the prompt template. The user context information may include information indicating at least one of the location where the user is located, the region where the device that acquired the utterance is located at said location, the type of the device that acquired the utterance, and the time at which the utterance was acquired. The device list may include device context information for each device. The device context information may include information indicating at least one of a description of the device corresponding to the source of the device context information, the type of the device, the location where the device is located, and the specific region where the device is located.

[0139] According to one embodiment, a method for operating an electronic device (e.g., a master device (320)) is provided. The method may include: acquiring a user’s utterance through a microphone of the electronic device or a communication circuit of the electronic device; generating a first prompt to be input into a first LLM (large language model) based on the utterance and a list of devices registered in the electronic device; checking an update target in the list of devices and an update content of a second prompt to be input into a second LLM related to the update target from an output value output from the first LLM as a result of inputting the first prompt into the first LLM; and transmitting the update content to an external electronic device configured to generate the second prompt through the communication circuit.

[0140] In the above explanation, prefixes such as “first,” “second,” and “third” are intended merely to distinguish components of the same name and are not assigned any special meaning in themselves, such as importance or order.

[0141] In the present disclosure, the expression 'connection' means not only a direct connection between components, but also an electrical connection where other components (e.g., resistors, inductors, etc.) are present between components.

[0142] The electronic device according to the various embodiments disclosed in this document may be of various forms. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a consumer electronics device. The electronic device according to the embodiments of this document is not limited to the devices described above.

[0143] The various embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In this document, phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B and C," and "at least one of A, B, or C" may each include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used simply to distinguish said components from other said components and do not limit said components in any other aspect (e.g., importance or order). Where any (e.g., 1st) component is referred to as "coupled" or "connected" to another (e.g., 2nd) component, with or without the terms "functionally" or "communicationly," it means that said any component may be connected to said other component directly (e.g., via a wire), wirelessly, or through a third component.

[0144] The term “module” as used in the various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. According to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0145] Various embodiments of the present document may be implemented as software (e.g., program (140)) comprising one or more instructions stored in a storage medium (e.g., internal memory (136) or external memory (138)) readable by a machine (e.g., electronic device (101)). For example, a processor (e.g., processor (120)) of the machine (e.g., electronic device (101)) may call at least one of the one or more instructions stored in the storage medium and execute it. This enables the machine to be operated to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by the machine may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily.

[0146] According to one embodiment, the method according to the various embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or an application store (e.g., Play Store). TM It can be distributed online (e.g., downloaded or uploaded) through ) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0147] According to various embodiments, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to various embodiments, one or more of the components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to integration. According to various embodiments, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In an electronic device, mike; Communication circuit for communication with external electronic devices; Memory for storing instructions; and It includes at least one processor, When the above instruction is executed individually or collectively by the at least one processor, the electronic device, Acquiring the user's speech through the above microphone or the above communication circuit, and Information regarding at least one control command for controlling at least one external electronic device is received from the at least one external electronic device through the communication circuit, and Based on the received information and the utterance, a first prompt to be input into a first LLM (large language model) is generated, and As a result of inputting the first prompt into the first LLM, checking the update target of the at least one external electronic device and the update content of the second prompt to be input to the second LLM associated with the update target from the output value output from the first LLM, and An electronic device configured to transmit the above update content to an external electronic device configured to generate the above second prompt through the above communication circuit.

2. In claim 1, when the instruction is executed individually or collectively by the at least one processor, the electronic device, The first prompt is generated by injecting information regarding the utterance and at least one control command into a pre-prepared prompt template, and An electronic device in which the above prompt template includes instructions indicating what the first LLM must do.

3. In claim 2, when the instruction is executed individually or collectively by the at least one processor, the electronic device, User context information related to the utterance is obtained through the microphone or the communication circuit, and The first prompt is generated by injecting the above utterance, information regarding the above at least one control command, and the above user context information into the prompt template, and The above user context information comprises information indicating at least one of the following: a location where at least one external electronic device is located, a region where a device that acquired the utterance is located at said location, a type of device that acquired the utterance, and a time at which the utterance was acquired.

4. In claim 3, when the instruction is executed individually or collectively by the at least one processor, the electronic device, Device context information for each of the above-mentioned at least one external electronic device is received from the above-mentioned at least one external electronic device through the communication circuit, and The first prompt is generated in the prompt template using the above utterance, the above user context information, the information regarding the at least one control command, and the device context information for each of the at least one external electronic device. An electronic device comprising information indicating at least one of a description of a device corresponding to the source of the device context information, a type of the device, a place where the device is located, and a specific region where the device is located.

5. An electronic device comprising the first LLM in any one of claims 1 to 4, wherein the memory comprises the first LLM.

6. In any one of claims 1 to 4, when the instruction is executed individually or collectively by the at least one processor, the electronic device, The above first prompt is transmitted to an external electronic device having the above first LLM through the above communication circuit, and An electronic device that receives the output value through the above communication circuit.

7. In a method of operating an electronic device, The operation of obtaining a user's speech through a microphone of the electronic device or a communication circuit of the electronic device; The operation of receiving information regarding at least one control command for controlling at least one external electronic device from the at least one external electronic device through the communication circuit; An operation to generate a first prompt to be input into a first LLM (large language model) based on the received information and the utterance; An operation of confirming, from the output value output from the first LLM as a result of inputting the first prompt into the first LLM, the update content of the second prompt to be put into the second LLM associated with the update target and the update target in the at least one external electronic device; and A method comprising the operation of transmitting the above update content to an external electronic device configured to generate the second prompt through the above communication circuit.

8. In claim 7, the operation of generating the first prompt is, The operation of generating the first prompt by injecting information regarding the utterance and the at least one control command into a pre-prepared prompt template, and A method in which the above prompt template includes instructions indicating what the above first LLM must do.

9. In claim 8, the operation of obtaining user context information related to the utterance through the microphone or the communication circuit is further included, The operation of generating the first prompt includes the operation of generating the first prompt by injecting information regarding the utterance, the at least one control command, and the user context information into the prompt template. A method comprising information indicating at least one of the following: the location where the at least one external electronic device is located, the region where the device that acquired the utterance is located at the location, the type of the device that acquired the utterance, and the time at which the utterance was acquired.

10. In claim 9, the operation of receiving device context information for each of the at least one external electronic device from the at least one external electronic device through the communication circuit is further included. The operation of generating the first prompt includes generating the first prompt in the prompt template, information regarding the utterance, the at least one control command, the user context information, and the device context information for each of the at least one external electronic device. A method comprising information indicating at least one of a description of a device corresponding to the source of the device context information, a type of device, a place where the device is located, and a specific region where the device is located.

11. In an electronic device, mike; Communication circuit for communication with external electronic devices; Memory for storing instructions; and It includes at least one processor, When the above instruction is executed individually or collectively by the at least one processor, the electronic device, Acquiring the user's speech through the above microphone or the above communication circuit, and Based on the above utterance and the list of devices registered in the electronic device, a first prompt to be input into a first LLM (large language model) is generated, and As a result of inputting the first prompt into the first LLM, checking the update target in the device list and the update content of the second prompt to be input into the second LLM related to the update target from the output value output from the first LLM, and An electronic device configured to transmit the above update content to an external electronic device configured to generate the above second prompt through the above communication circuit.

12. In claim 11, when the instruction is executed individually or collectively by the at least one processor, the electronic device, The first prompt is generated by injecting the utterance and the device list into a pre-prepared prompt template, and An electronic device in which the above prompt template includes instructions indicating what the first LLM must do.

13. In claim 12, when the instruction is executed individually or collectively by the at least one processor, the electronic device, User context information related to the utterance is obtained through the microphone or the communication circuit, and By injecting the above utterance, the above device list, and the above user context information into the above prompt template, the first prompt is generated, and The above user context information is an electronic device comprising information indicating at least one of the location where the user is located, the region where the device that acquired the utterance at the location is located, the type of the device that acquired the utterance, and the time at which the utterance was acquired.

14. In claim 13, the device list includes device context information for each device, and An electronic device comprising information indicating at least one of a description of a device corresponding to the source of the device context information, a type of the device, a place where the device is located, and a specific region where the device is located.

15. In a method of operating an electronic device, The operation of obtaining a user's speech through a microphone of the electronic device or a communication circuit of the electronic device; An operation to generate a first prompt to be input into a first LLM (large language model) based on the above utterance and the list of devices registered in the electronic device; An operation to check the update target in the device list and the update content of the second prompt to be input to the second LLM related to the update target, from the output value output from the first LLM as a result of inputting the first prompt into the first LLM; and A method comprising the operation of transmitting the above update content to an external electronic device configured to generate the second prompt through the above communication circuit.

Images