Electronic device for providing result to user utterance, operating method thereof, and non-transitory storage medium

Abstract

This electronic device may comprise: a memory storing instructions and including at least one storage medium; and at least one processor including processing circuitry. The memory may store user data related to a user of the electronic device, and a plurality of prompt templates. The electronic device may: receive a user input including a query for an artificial intelligence (AI) model; convert the user input into text; input the converted text into a first artificial intelligence model trained on information about at least one domain; and input a prompt requesting the text to be divided into semantic units. The electronic device may: provide, to the second artificial intelligence model, information about the state and applications of the electronic device as an input; determine a domain and a user intent on the basis of an output result from the second artificial intelligence model; and control output of a response to the user input.

Description

An electronic device that provides a result for user utterance, a method of operation thereof, and a non-transient storage medium

[0001] This document discloses an electronic device that provides results for user utterances, a method of operation thereof, and a non-transient storage medium.

[0002] With the development of digital technology, various types of electronic devices such as mobile communication terminals, PDAs (personal digital assistants), electronic notebooks, smartphones, tablet PCs (personal computers), or wearable devices are widely used. To support and enhance the functionality of these electronic devices, the hardware and / or software parts of the devices are continuously being improved.

[0003] Recently, technology capable of understanding language through Large Language Models (LLMs) and summarizing or abbreviating long sentences has been advancing. Electronic devices acquire result data regarding user inquiries via LLM servers and provide this data to the user. However, since the result data contains vast amounts of information, users may need to examine the data to find the specific information they need.

[0004] Large Language Models (LLMs) are language models recently developed by training on vast amounts of text data. By sophisticatedly learning human language patterns, they can achieve higher language comprehension capabilities than existing models. They can provide various functions that traditional models could not perform and are currently utilized in diverse fields such as text generation, translation, question answering, and text classification. Text classification is a branch of Natural Language Processing (NLP) that refers to the process of dividing given text into predefined classes or categories using machine learning. Because LLMs possess higher language comprehension capabilities than existing models, they demonstrate superior performance (accuracy) in text classification as well. The commands input to an LLM are called prompts. Prompts do not have a fixed format and can be provided in text form. LLMs can receive prompts and output results by predicting the sentences that follow.

[0005] A voice assistant receives voice input, converts it into text, and can understand the intent from the converted text. The electronic device may include automatic speech recognition (ASR) for converting voice to text, natural language understanding (NLU) for understanding the intent of the text, text-to-speech (TTS) used to output a response, and a client for communicating with a server from other electronic devices (e.g., terminals) and delivering necessary actions to each app.

[0006] The information described above may be provided as related art for the purpose of aiding understanding of the present disclosure. No claim or determination is made as to whether any of the foregoing may be applied as prior art related to the present disclosure.

[0007] Since NLU identifies intent through text, performance may degrade if noise (unintended words) caused by external sounds other than the user's utterance is mixed in, as it fails to accurately determine the intent. Additionally, when multiple commands are requested at once, such as in complex instructions, the system may fail to properly grasp the intent, leading to issues such as executing only some commands or malfunctions.

[0008] Even when a user says "Lower the volume," voice or noise may be mixed in with the input in a noisy environment. For example, an electronic device may recognize the text "Lower the volume" mixed with noise or external sounds generated by external sounds other than the user's speech. In this case, an incorrect speech may be transmitted to the generative AI, leading to incorrect response processing.

[0009] The electronic device according to this document can perform accurate response processing even for user utterances containing noise by first classifying the given text into intent units.

[0010] The electronic device may include a memory that stores instructions and includes one or more storage media, and at least one processor that includes processing circuitry. The memory stores user data related to a user of an electronic device and a plurality of prompt templates, and when instructions are executed individually or collectively by at least one processor, the electronic device receives user input including a query to an artificial intelligence (AI) model, converts the user input into text, inputs the converted text to a first artificial intelligence model learned with information on at least one domain, inputs a prompt requesting to divide into semantic units, classifies the input text into semantic units to determine the probability of being included in a specific domain corresponding to a specific topic among multiple domains, determines that the input text is included in a specific domain based on the probability of being included in a specific domain exceeding a specified level, transmits the converted text and information on the specific domain to a domain classifier, and provides the converted text, user data, the state of the electronic device, and information on the application as input to a second artificial intelligence model based on the fact that the probability of being included in a specific domain corresponding to a specific topic among multiple domains is less than a specified level as a result of classification by semantic unit, and determines the domain and user intent based on the output result of the second artificial intelligence model, thereby the user It can be controlled to output a response to the input.

[0011] A computer-readable non-transient storage medium storing one or more programs including instructions executable by at least one processor of an electronic device receives user input including a query to an artificial intelligence (AI) model, converts the user input into text, inputs the converted text into a first artificial intelligence model that has learned information about at least one domain, inputs a prompt requesting to divide into semantic units, classifies the input text into semantic units to determine the probability of being included in a specific domain corresponding to a specific topic among multiple domains, determines that the input text is included in a specific domain based on the probability of being included in a specific domain exceeding a specified level, transmits the converted text and information about the specific domain to a domain classifier, provides the converted text and information about user data, the state of the electronic device and the application as input to a second artificial intelligence model based on the result of classification into semantic units that the probability of being included in a specific domain corresponding to a specific topic among multiple domains is less than a specified level, and controls to determine the domain and user intent based on the output result of the second artificial intelligence model and output a response to the user input.

[0012] The electronic device according to the various embodiments of this document can process utterances containing noise because it first classifies a given text into intent units.

[0013] The electronic device according to the various embodiments of this document undergoes a process of inferring a more suitable command by comprehensively determining user usage patterns or the status of connected devices when a command is divided into multiple intent units. Through this, the electronic device can process commands that it could not previously process, thereby achieving better performance than before.

[0014] In relation to the description of the drawings, the same or similar reference numerals may be used for identical or similar components.

[0015] FIG. 1 is a block diagram of an electronic device in a network environment according to various embodiments.

[0016] FIG. 2 is a block diagram showing an integrated intelligence system according to various embodiments.

[0017] FIG. 3 is a diagram showing the form in which relationship information between concepts and operations is stored in a database according to various embodiments.

[0018] FIG. 4 is a block diagram of a generative artificial intelligence system according to one embodiment.

[0019] FIG. 5 is a block diagram illustrating the process of an electronic device outputting a response based on user input according to one embodiment.

[0020] FIG. 6 illustrates the configuration of a second artificial intelligence model according to one embodiment.

[0021] FIG. 7a illustrates a first embodiment in which an electronic device according to one embodiment outputs a response based on user input.

[0022] FIG. 7b illustrates a second embodiment in which an electronic device according to one embodiment outputs a response based on user input.

[0023] FIG. 8 illustrates a user interface in which an electronic device according to one embodiment outputs a response based on user input.

[0024] FIG. 9 is a flowchart illustrating the operation method of an electronic device according to one embodiment.

[0025] FIG. 10 is a flowchart illustrating the operation method of an electronic device according to one embodiment.

[0026] Hereinafter, embodiments of the present disclosure are described in detail with reference to the drawings so that those skilled in the art can easily practice them. However, the present disclosure may be embodied in various different forms and is not limited to the embodiments described herein. In relation to the description of the drawings, the same or similar reference numerals may be used for identical or similar components. Furthermore, in the drawings and related descriptions, descriptions of well-known functions and configurations may be omitted for clarity and brevity.

[0027] FIG. 1 is a block diagram of an electronic device (101) in a network environment (100) according to various embodiments. Referring to FIG. 1, in the network environment (100), the electronic device (101) may communicate with an electronic device (102) through a first network (198) (e.g., a short-range wireless communication network) or may communicate with at least one of an electronic device (104) or a server (108) through a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) through a server (108). According to one embodiment, the electronic device (101) may include a processor (120), memory (130), input module (150), sound output module (155), display module (160), audio module (170), sensor module (176), interface (177), connection terminal (178), haptic module (179), camera module (180), power management module (188), battery (189), communication module (190), subscriber identification module (196), or antenna module (197). In some embodiments, at least one of these components (e.g., connection terminal (178)) may be omitted from the electronic device (101), or one or more other components may be added. In some embodiments, some of these components (e.g., sensor module (176), camera module (180), or antenna module (197)) may be integrated into a single component (e.g., display module (160)).

[0028] The processor (120) can control at least one other component (e.g., hardware or software component) of the electronic device (101) connected to the processor (120) by executing software (e.g., program (140)), and can perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor (120) can store commands or data received from other components (e.g., sensor module (176) or communication module (190)) in volatile memory (132), process the commands or data stored in volatile memory (132), and store the resulting data in non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., central processing unit or application processor) or an auxiliary processor (123) that can operate independently or together with it (e.g., graphics processing unit, neural processing unit (NPU), image signal processor, sensor hub processor, or communication processor). For example, if the electronic device (101) includes a main processor (121) and an auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a designated function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as part thereof.

[0029] The auxiliary processor (123) may control at least some of the functions or states associated with at least one component of the electronic device (101) (e.g., display module (160), sensor module (176), or communication module (190)) on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. According to one embodiment, the auxiliary processor (123) (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module (180) or communication module (190)). According to one embodiment, the auxiliary processor (123) (e.g., neural network processing unit) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or through a separate server (e.g., server (108)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers.An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may include a software structure, either additionally or substantially.

[0030] The memory (130) can store various data used by at least one component of the electronic device (101) (e.g., processor (120) or sensor module (176)). The data may include, for example, input data or output data for software (e.g., program (140)) and related commands. The memory (130) may include volatile memory (132) or non-volatile memory (134).

[0031] The program (140) may be stored as software in memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0032] The input module (150) can receive commands or data to be used for a component of the electronic device (101) (e.g., processor (120)) from outside the electronic device (101) (e.g., user). The input module (150) may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0033] The sound output module (155) can output a sound signal to the outside of the electronic device (101). The sound output module (155) may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as multimedia playback or recording playback. The receiver may be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part thereof.

[0034] The display module (160) can visually provide information to an external (e.g., user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling said device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of the force generated by said touch.

[0035] The audio module (170) can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150) or output sound through the sound output module (155) or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphones) connected directly or wirelessly to the electronic device (101).

[0036] The sensor module (176) can detect the operating state of the electronic device (101) (e.g., power or temperature) or the external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, the sensor module (176) may include, for example, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0037] The interface (177) may support one or more specified protocols that can be used for the electronic device (101) to be connected directly or wirelessly to an external electronic device (e.g., electronic device (102)). According to one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0038] The connection terminal (178) may include a connector through which the electronic device (101) can be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0039] The haptic module (179) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that the user can perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module (179) may include, for example, a motor, a piezoelectric element, or an electric stimulation device.

[0040] The camera module (180) can capture still images and video. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0041] The power management module (188) can manage the power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least part of a power management integrated circuit (PMIC).

[0042] The battery (189) can supply power to at least one component of the electronic device (101). According to one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0043] The communication module (190) can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between an electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may include one or more communication processors that operate independently of the processor (120) (e.g., application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., cellular communication module, short-range wireless communication module, or GNSS (global navigation satellite system) communication module) or a wired communication module (194) (e.g., LAN (local area network) communication module, or power line communication module). The corresponding communication module among these communication modules can communicate with an external electronic device (104) through a first network (198) (e.g., a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data association)) or a second network (199) (e.g., a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can identify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) using subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module (196).

[0044] The wireless communication module (192) can support 5G networks and next-generation communication technologies following 4G networks, for example, new radio access technology. NR access technology can support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and connection of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low-latency communications (URLLC)). The wireless communication module (192) can support a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate, for example. The wireless communication module (192) can support various technologies for securing performance in the high-frequency band, such as beamforming, massive MIMO (multiple-input and multiple-output), full-dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large-scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), external electronic device (e.g., electronic device (104)), or network system (e.g., second network (199)). According to one embodiment, the wireless communication module (192) can support a Peak data rate (e.g., 20 Gbps or more) for realizing eMBB, loss coverage (e.g., 164 dB or less) for realizing mMTC, or U-plane latency (e.g., downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less) for realizing URLLC.

[0045] An antenna module (197) can transmit a signal or power to or from an external source (e.g., an external electronic device). According to one embodiment, the antenna module (197) may include an antenna comprising a radiator made of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as a first network (198) or a second network (199), may be selected from the plurality of antennas, for example, by a communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, other components (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module (197).

[0046] According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the printed circuit board and capable of supporting a specified high frequency band (e.g., mmWave band), and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top surface or side surface) of the printed circuit board and capable of transmitting or receiving a signal of the specified high frequency band.

[0047] At least some of the above components can be connected to each other via a communication method between peripheral devices (e.g., bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)) and exchange signals (e.g., commands or data) with each other.

[0048] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) through a server (108) connected to a second network (199). Each of the external electronic devices (102, or 104) may be the same or different type of device as the electronic device (101). According to one embodiment, all or part of the operations performed on the electronic device (101) may be performed on one or more of the external electronic devices (102, 104, or 108). For example, if the electronic device (101) needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device (101) may request one or more external electronic devices to perform at least part of the function or service instead of performing the function or service itself or additionally. One or more external electronic devices that receive the above request may execute at least part of the requested function or service, or additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may provide the result as is or additionally processed as at least part of the response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device (101) may provide ultra-low latency services using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server using machine learning and / or neural networks. According to one embodiment, the external electronic device (104) or the server (108) may be included within a second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0049] FIG. 2 is a block diagram showing an integrated intelligence system according to various embodiments.

[0050] Referring to FIG. 2, according to one embodiment, the integrated intelligent system may include an electronic device (210) (e.g., the electronic device (101) of FIG. 1), an intelligent server (230) (e.g., the server (108) of FIG. 1), and a service server (250) (e.g., the server (108) of FIG. 1).

[0051] According to one embodiment, the electronic device (210) may be a terminal device (or electronic device) capable of connecting to the Internet, and may be, for example, a mobile phone, a smartphone, a PDA (personal digital assistant), a laptop computer, a TV (television), a white goods appliance, a wearable device, a HMD (head mounted display), or a smart speaker.

[0052] According to the illustrated embodiment, the electronic device (210) may include a communication interface (213) (e.g., interface (177) of FIG. 1), a microphone (212) (e.g., input module (150) of FIG. 1), a speaker (216) (e.g., sound output module (155) of FIG. 1), a display module (211) (e.g., display module (160) of FIG. 1), a memory (215) (e.g., memory (130) of FIG. 1), or a processor (214) (e.g., processor (120) of FIG. 1). The listed components may be operatively or electrically connected to each other. The electronic device (210) may include at least some of the configurations and / or functions of the electronic device (101) of FIG. 1.

[0053] According to one embodiment, the communication interface (213) may be configured to be connected to an external device to transmit and receive data. According to one embodiment, the microphone (212) may receive sound (e.g., user speech) and convert it into an electrical signal. According to one embodiment, the speaker (216) may output the electrical signal as sound (e.g., voice).

[0054] According to one embodiment, the display module (211) may be configured to display an image or video. According to one embodiment, the display module (211) may also display a graphic user interface (GUI) of an app (or application program) being executed. The display module (211) of one embodiment may receive touch input through a touch sensor. For example, the display module (211) may receive text input through a touch sensor in an image keyboard area displayed within the display module (211).

[0055] According to one embodiment, memory (215) can store a client module (218), an SDK (software development kit) (217), and a plurality of apps (219a, 219b). The client module (218) and the SDK (217) can form a framework (or solution program) for performing general-purpose functions. Additionally, the client module (218) or the SDK (217) can form a framework for processing user input (e.g., voice input, text input, touch input).

[0056] According to one embodiment, the plurality of apps (219a, 219b) stored in memory (215) may be programs for performing a designated function. According to one embodiment, the plurality of apps may include a first app (219a) and a second app (219b). According to one embodiment, each of the plurality of apps (219a, 219b) may include a plurality of operations for performing a designated function. For example, the apps (219a, 219b) may include an alarm app, a message app, and / or a schedule app. According to one embodiment, the plurality of apps (219a, 219b) may be executed by a processor (214) to sequentially execute at least some of the plurality of operations.

[0057] According to one embodiment, the processor (214) can control the overall operation of the electronic device (210). For example, the processor (214) can be electrically connected to a communication interface (213), a microphone (212), a speaker (216), and a display module (211) to perform a specified operation.

[0058] The number of processors (214) may be one or more. For example, the processor (214) may have the structure of a multi-core processor such as a dual core, quad core, or hexa core.

[0059] The processor (214) can control the operations of the electronic device (101) by executing instructions stored in memory (215). For example, the processor (214) may correspond to a plurality of processors that divide and collectively perform a plurality of operations among the processors.

[0060] According to one embodiment, the processor (214) may also perform a specified function by executing a program stored in the memory (215). For example, the processor (214) may execute at least one of the client module (218) or the SDK (217) to perform the following operations for processing user input. The processor (214) may, for example, control the operation of a plurality of apps (219a, 219b) through the SDK (217). The following operations described as the operation of the client module (218) or the SDK (217) may be operations performed by the execution of the processor (214).

[0061] According to one embodiment, the client module (218) can receive user input. For example, the client module (218) can receive a voice signal corresponding to a user utterance detected through a microphone (212). Alternatively, the client module (218) can receive touch input detected through a display module (211). Alternatively, the client module (218) can receive text input detected through a keyboard or a virtual keyboard. In addition, various forms of user input detected through an input module included in the electronic device (210) or an input module connected to the electronic device (210) can be received. The client module (218) can transmit the received user input to an intelligent server (230). Along with the received user input, the client module (218) can transmit status information of the electronic device (210) to the intelligent server (230). The status information may be, for example, execution status information of an app.

[0062] According to one embodiment, the client module (218) can receive a result corresponding to the received user input. For example, the client module (218) can receive a result corresponding to the received user input if the intelligent server (230) can produce a result corresponding to the received user input. The client module (218) can display the received result on the display module (211). Additionally, the client module (218) can output the received result as audio through the speaker (216).

[0063] According to one embodiment, the client module (218) may receive a plan corresponding to the received user input. The client module (218) may display the results of executing a plurality of actions of the app according to the plan on the display module (211). The client module (218) may, for example, sequentially display the results of executing a plurality of actions on the display module (211) and output audio through the speaker (216). The electronic device (210) may, for another example, display only some of the results of executing a plurality of actions (e.g., the result of the last action) on the display module (211) and output audio through the speaker (216).

[0064] According to one embodiment, the client module (218) may receive a request from the intelligent server (230) to obtain information necessary to produce a result corresponding to a voice input. According to one embodiment, the client module (218) may transmit the necessary information to the intelligent server (230) in response to the request.

[0065] According to one embodiment, the client module (218) can transmit result information of executing a plurality of operations according to a plan to the intelligent server (230). The intelligent server (230) can use the result information to confirm that the received user input has been processed correctly.

[0066] According to one embodiment, the client module (218) may include a voice recognition module. According to one embodiment, the client module (218) may recognize voice input that performs a limited function through the voice recognition module. For example, the client module (218) may execute an intelligent app for processing voice input to perform an organic action through a specified input (e.g., Wake Up!).

[0067] According to one embodiment, an intelligent server (230) can receive information related to user voice input from an electronic device (210) via a communication network. According to one embodiment, the intelligent server (230) can convert data related to the received voice input into text data. According to one embodiment, the intelligent server (230) can generate a plan for performing a task corresponding to the user voice input based on the text data.

[0068] According to one embodiment, a plan may be generated by an artificial intelligence (AI) system. The AI ​​system may be a rule-based system or a neural network-based system (e.g., a feedforward neural network (FNN), a recurrent neural network (RNN)). Alternatively, it may be a combination of the foregoing or a different AI system. According to one embodiment, the plan may be selected from a set of predefined plans or may be generated in real time in response to a user request. For example, the AI ​​system may select at least one plan from a plurality of predefined plans.

[0069] According to one embodiment, the intelligent server (230) may transmit the result according to the generated plan to the electronic device (210) or transmit the generated plan to the electronic device (210). According to one embodiment, the electronic device (210) may display the result according to the plan on the display module (211). According to one embodiment, the electronic device (210) may display the result of executing the operation according to the plan on the display module (211).

[0070] According to one embodiment, the intelligent server (230) may include a front end (231), a natural language platform (232), a capsule database (238), an execution engine (233), an end user interface (234), a management platform (235), a big data platform (236), or an analytic platform (237).

[0071] According to one embodiment, the front end (231) can receive user input received from the electronic device (210). The front end (231) can transmit a response corresponding to the user input.

[0072] According to one embodiment, the natural language platform (232) may include an automatic speech recognition module (ASR module) (232a), a natural language understanding module (NLU module) (232b), a planner module (232c), a natural language generator module (NLG module) (232d), or a text to speech module (TTS module) (232e).

[0073] According to one embodiment, an automatic speech recognition module (232a) can convert voice input received from an electronic device (210) into text data. According to one embodiment, a natural language understanding module (232b) can identify the user's intent using the text data of the voice input. For example, the natural language understanding module (232b) can identify the user's intent by performing a syntactic analysis or a semantic analysis on the user input in the form of text data. According to one embodiment, the natural language understanding module (232b) can identify the meaning of a word extracted from the voice input using linguistic features (e.g., grammatical elements) of a morpheme or phrase, and determine the user's intent by matching the identified meaning of the word to the intent. The natural language understanding module (223b) can acquire intent information corresponding to the user's utterance. The intent information may be information indicating the user's intent determined by interpreting the text data. The intent information may include information indicating an action or function that the user intends to execute using the device.

[0074] According to one embodiment, the planner module (232c) can generate a plan using the intent and parameters determined by the natural language understanding module (232b). According to one embodiment, the planner module (232c) can determine a plurality of domains necessary to perform a task based on the determined intent. The planner module (232c) can determine a plurality of actions included in each of the plurality of domains determined based on the intent. According to one embodiment, the planner module (232c) can determine parameters necessary to execute the determined plurality of actions or result values ​​output by the execution of the plurality of actions. The parameters and the result values ​​may be defined as concepts of a specified format (or class). Accordingly, the plan may include a plurality of actions and a plurality of concepts determined by the user's intent. The planner module (232c) can determine the relationship between the plurality of actions and the plurality of concepts in a stepwise (or hierarchical) manner. For example, the planner module (232c) can determine the execution order of multiple actions determined based on the user's intentions based on multiple concepts. In other words, the planner module (232c) can determine the execution order of multiple actions based on parameters required for the execution of multiple actions and results output by the execution of multiple actions. Accordingly, the planner module (232c) can generate a plan that includes association information (e.g., ontology) between multiple actions and multiple concepts. The planner module (232c) can generate the plan using information stored in a capsule database in which a set of relationships between concepts and actions is stored.

[0075] According to one embodiment, the natural language generation module (232d) can change specified information into a text form. The information changed into a text form may be in the form of a natural language utterance. According to one embodiment, the text-to-speech conversion module (232e) can change information in a text form into information in a speech form.

[0076] According to one embodiment, some or all functions of the natural language platform (232) may also be implemented in an electronic device (210).

[0077] The above capsule database can store information regarding the relationships between multiple concepts and actions corresponding to multiple domains. A capsule according to one embodiment may include multiple action objects (or action information) and concept objects (or concept information) included in a plan. According to one embodiment, the capsule database can store multiple capsules in the form of a CAN (concept action network). According to one embodiment, multiple capsules may be stored in a function registry included in the capsule database.

[0078] The above capsule database may include a strategy registry that stores strategy information necessary for determining a plan corresponding to user input. The strategy information may include reference information for determining one plan when there are multiple plans corresponding to user input. According to one embodiment, the capsule database may include a follow-up registry that stores information on a follow-up action for suggesting a follow-up action to the user in a specified situation. The follow-up action may include, for example, a follow-up utterance. According to one embodiment, the capsule database may include a layout registry that stores layout information of information output through an electronic device (210). According to one embodiment, the capsule database may include a vocabulary registry that stores vocabulary information included in the capsule information. According to one embodiment, the capsule database may include a dialogue registry that stores information on a conversation (or interaction) with the user. The capsule database may update stored objects through a developer tool. The above developer tool may include, for example, a function editor for updating action objects or concept objects. The above developer tool may include a vocabulary editor for updating vocabulary. The above developer tool may include a strategy editor for creating and registering strategies for determining plans. The above developer tool may include a dialogue editor for creating conversations with a user.The above developer tool may include a follow-up editor capable of activating a follow-up goal and editing a follow-up utterance that provides a hint. The follow-up goal may be determined based on a currently set goal, user preferences, or environmental conditions. In one embodiment, the capsule database may also be implemented within an electronic device (210).

[0079] According to one embodiment, the execution engine (233) can produce a result using the generated plan. The end user interface (234) can transmit the produced result to the electronic device (210). Accordingly, the electronic device (210) can receive the result and provide the received result to the user. According to one embodiment, the management platform (235) can manage information used in the intelligent server (230). According to one embodiment, the big data platform (236) can collect user data. According to one embodiment, the analysis platform (237) can manage the quality of service (QoS) of the intelligent server (230). For example, the analysis platform (237) can manage the components and processing speed (or efficiency) of the intelligent server (230).

[0080] According to one embodiment, the service server (250) may provide a service designated to the electronic device (210), such as ordering food or booking a hotel. According to one embodiment, the service server (250) may be a server operated by a third party. According to one embodiment, the service server (250) may provide information to the intelligent server (230) for generating a plan corresponding to a received voice input. The provided information may be stored in a capsule database. Additionally, the service server (250) may provide result information according to the plan to the intelligent server (230). The service server (250) may include a plurality of service providers (e.g., CP service A (251), CP service B (252), CP service C (253)), and each service provider (251, 252, 253) may provide a function for a domain associated with each capsule stored in the capsule database (238) of the intelligent server (230).

[0081] In the integrated intelligent system described above, the electronic device (210) can provide various intelligent services to the user in response to user input. The user input may include, for example, input via a physical button, touch input, or voice input.

[0082] According to one embodiment, the electronic device (210) may provide a voice recognition service through an intelligent app (or voice recognition app) stored internally. In this case, for example, the electronic device (210) may recognize a user utterance or voice input received through the microphone (212) and provide a service to the user corresponding to the recognized voice input.

[0083] According to one embodiment, the electronic device (210) may perform a specified action based on a received voice input, either alone or in conjunction with the intelligent server (230) and / or service server (250). For example, the electronic device (210) may execute an app corresponding to the received voice input and perform a specified action through the executed app.

[0084] According to one embodiment, when an electronic device (210) provides services together with an intelligent server (230) and / or a service server (250), the electronic device (210) can detect user speech using the microphone (212) and generate a signal (or voice data) corresponding to the detected user speech. The electronic device (210) can transmit the voice data to the intelligent server (230) via a network (240) using a communication interface (213).

[0085] An intelligent server (230) according to one embodiment may generate, in response to a voice input received from an electronic device (210), a plan for performing a task corresponding to the voice input, or a result of performing an operation according to the plan. The plan may include, for example, a plurality of operations for performing a task corresponding to a user's voice input, and a plurality of concepts related to the plurality of operations. The concepts may define parameters input to the execution of the plurality of operations or result values ​​output by the execution of the plurality of operations. The plan may include association information between the plurality of operations and the plurality of concepts.

[0086] According to one embodiment, the electronic device (210) can receive the response using a communication interface (213). The electronic device (210) can output a voice signal generated inside the electronic device (210) to the outside using the speaker (216), or output an image generated inside the electronic device (210) to the outside using a display module (211).

[0087] In FIG. 2, an example is described in which voice recognition of user input received from an electronic device (210), natural language understanding and generation, and output of results using a plan are performed on an intelligent server (230), but various embodiments of this document are not limited thereto. For example, at least some components of the intelligent server (230) (e.g., natural language platform (232), execution engine (233), capsule database (238)) may be embedded in the electronic device (210) (or the electronic device (101) of FIG. 1), and the operation may be performed by the electronic device (210).

[0088] FIG. 3 is a diagram showing the form in which relationship information between concepts and operations is stored in a database according to various embodiments.

[0089] According to one embodiment, a capsule database (e.g., capsule database (238) of FIG. 2) of an intelligent server (e.g., intelligent server (230) of FIG. 2) may store capsules in the form of a CAN (concept action network) (300). The capsule database may store actions for processing tasks corresponding to user voice input, and parameters required for said actions, in the form of a CAN (concept action network).

[0090] According to one embodiment, the capsule database may store a plurality of capsules (capsule (A) (310), capsule (B) (320)) corresponding to each of a plurality of domains (e.g., applications). According to one embodiment, one capsule (e.g., capsule (A) (310)) may correspond to one domain (e.g., location (geo), application). Additionally, one capsule may correspond to at least one service provider (e.g., CP 1 (331) or CP 2 (332)) for performing functions for the domain associated with the capsule. According to one embodiment, one capsule may include at least one operation (350) and at least one concept (360) for performing a designated function.

[0091] According to one embodiment, a natural language platform (e.g., the natural language platform (232) of FIG. 2) can generate a plan for performing a task corresponding to a received voice input using capsules stored in a capsule database. For example, a planner module of the natural language platform (e.g., the planner module (232c) of FIG. 2) can generate a plan using capsules stored in a capsule database. For example, a plan can be generated using the actions (311, 313) and concepts (312, 314) of capsule A (310) and the actions (321) and concepts (322) of capsule B (320).

[0092] FIG. 4 is a block diagram of a generative artificial intelligence system according to one embodiment.

[0093] Referring to FIG. 4, the generative artificial intelligence system (400) may include a generative AI model (450), an AI framework (440), a user query / response interface (410), an application / service component (430), and a knowledge repository (420). The AI ​​framework (440) may include a prompt design component (442), an API / plugin management component (444), and an output modification component (446).

[0094] According to one embodiment, a user query / response interface (410) may receive user input. The user input may be in the form of natural language, images, and / or videos. Additionally, context information may be transmitted along with the user input. The context information may include various additional information at the time of user input. For example, the context information may include information related to the user or the electronic device, such as information about the application currently being used by the user or the user's location information. Additionally, the user input may be in a mixed form of the aforementioned natural language, images, sounds, and context information. Furthermore, the user input may be in a non-natural language form, such as selecting a menu.

[0095] According to one embodiment, the user question / response interface (410) can output results of the generative artificial intelligence system (400) to the user. The output may be in the form of natural language or specific content, and may also be provided in a form such as an action requested by the user.

[0096] According to one embodiment, the AI ​​framework (440) receives input from a user and can coordinate and control each component necessary to perform the user's intent based on the user's query.

[0097] According to one embodiment, user input received from a user question / response interface (410) may be transmitted to a prompt design component (442). The prompt design component (442) may be used to generate prompts suitable for inputting user input into a large language model (LLM) or a large multimodal model (LMM). The prompt design component (442) may be an AI component that uses machine learning algorithms or neural networks to develop better prompts over time. The prompt design component (442) may acquire and generate prompts by accessing a knowledge repository (420) containing user preference data, a prompt library, and prompt examples based on user input, and transmit the generated prompts to the LLM or LMM.

[0098] According to one embodiment, the API / Plug-in management component (444) can perform the role of communicating with external information when there is a request for additional information when transmitting user input as input to a generative model. The API / Plug-in management component (444) establishes a channel to communicate with the outside of the AI ​​interface via an API, and can enable access to various data sources through the established channel. Additionally, the API / Plug-in management component (444) can request an action via an API when an application or service needs to perform an action that executes the user input as a final step, rather than an intermediate result. Information obtained from the outside may be used to generate a prompt in the prompt design component (442) along with the user input, or it may be transmitted as input to the generative model.

[0099] According to one embodiment, an output modification component (446) (or refiner component) can fine-tune the output of a generative model. For example, the output modification component (446) can verify whether the content generated through the LLM and / or LMM is irrelevant, contains biased content, or contains harmful content. Additionally, the output modification component (446) can determine the extent to which the output matches what the user wants and, if additional processing is required, proceed with that process. Furthermore, the output modification component (446) can configure and provide the user with hints to avoid unwanted output.

[0100] According to one embodiment, a generative AI model (450) may generally refer to an artificial intelligence neural network that generates new forms of data based on user input information. The generative AI model (450) may include a model that generates images and / or a model that generates language. Models that generate images include, but are not limited to, GANs (generative adversarial networks) and VAEs (variational auto encoders), and examples of models that use VAEs and Diffusion-based generative models with Transformer structures can be cited. Models that generate language are models trained to output the most statistically appropriate output value based on input values, and examples of which include models such as CHAT-GPT 3 and CHAT-GPT 4. There are also LMMs (large multimodal models) that can recognize various forms of data input, such as text, images, and voice, and generate new data corresponding to them.

[0101] According to one embodiment, a generative artificial intelligence model (450) may be an artificial intelligence model that learns various data to generate new information and sentences. In this document, a generative artificial intelligence model may also be referred to as an AI model or a large language model (LM).

[0102] According to one embodiment, a user may input user input including a query to obtain desired information through an AI model. For example, the user may input the query via text input using a keypad or keyboard, or via voice input using a microphone. User input to the AI ​​model may be input to the AI ​​model as a prompt. The prompt serves as a command for generating a response in the AI ​​model and can induce the AI ​​model to perform the task desired by the user.

[0103] According to one embodiment, in order for a user to obtain a desired response through an AI model, it may be necessary to write a prompt that the AI ​​model can understand well and operate effectively through the AI ​​model. For example, parameters required for the AI ​​model's response include information such as reader level, response length, respondent's perspective, output format, and output language. If these parameters are written on the prompt through user input, the AI ​​model can clearly understand the user's query and provide the desired response. When a prompt containing various parameters is input into the AI ​​model (or LLM), the AI ​​model can output a response in English, in a conversational format, and within 500 characters at an elementary school level, as defined in the prompt. Since the AI ​​model can be used by various users, there may be differences in the ability / level to write prompts for the AI ​​model among individuals, and consequently, it may be difficult to write the optimal prompt for the AI ​​model intended for use.

[0104] FIG. 5 is a block diagram illustrating the process of an electronic device outputting a response based on user input according to one embodiment.

[0105] An electronic device (500) can receive user input and convert it into text using speech recognition technology (e.g., automatic speech recognition (ASR)). The electronic device (500) can transmit the converted text to a first artificial intelligence model (510). The first artificial intelligence model (510) may be a model that classifies the utterance intent of the input text. The first artificial intelligence model (510) may have a relatively lower cost compared to the second artificial intelligence model (520). Cost may refer to the resources consumed by the artificial intelligence model. The electronic device (500) can benefit in terms of cost by using the first artificial intelligence model (510), which has a relatively lower cost, to classify the intent of the user's utterance, and if classification fails, by using the second artificial intelligence model (520), which has a relatively higher cost. The first artificial intelligence model (510) is a relatively simple model that can perform the role of classifying the user's speech into a large category, such as what kind of request it is (e.g., a request to ask about the weather, a request to play music). The second artificial intelligence model (520) is a relatively more complex and accurate model and can be used when the first artificial intelligence model (510) fails to accurately grasp the user's intention.

[0106] If the electronic device (500) fails to classify the intent of the user's utterance, it can use a second artificial intelligence model (520) to reclassify the intent of the user's utterance. Classification failure may mean that the first artificial intelligence model (510) did not accurately understand the user's utterance and failed to match the user's intent to a domain. A situation where domain matching is incorrect may include, for example, a situation where the user asks, "What is the weather like today?" and the first artificial intelligence model (510) incorrectly classifies it into the category of "playing music." A domain may refer to a specific topic or field (e.g., medicine, law, sports, finance). For example, in the medical domain, words such as "disease," "symptom," and "treatment" appear frequently, and in the financial domain, words such as "stock," "investment," and "market" appear frequently. Based on these characteristics, the LLM can determine which domain the input text belongs to based on probability. When the domain is determined appropriately, the probability of correctly determining the intent of the user's utterance can increase.

[0107] The first artificial intelligence model (510) may be a lightweight LLM model with a number of parameters less than a specified level, or a deep learning model with a number of parameters less than a specified level. The first artificial intelligence model (510) can calculate the probability that the input text is included in a specific domain. The first artificial intelligence model (510) can calculate the probability that the input text belongs to a domain for each domain. Here, the input text may include text converted from a user's utterance. The electronic device (500) can determine the specific domain having the highest probability among the calculated probabilities. Using the first artificial intelligence model (510), the probability that the user utterance is included in a specific domain is checked, and if the probability exceeds a specified level, the intent of the user utterance can be matched with the specific domain. The electronic device (500) may not match the intent of the user utterance with the specific domain if the probability that the user utterance is included in a specific domain is less than a specified level. The electronic device (500) may determine that classification has failed if the probability that the user utterance is included in a specific domain is less than a specified level. The electronic device (500) can provide text input to a second artificial intelligence model (520) based on failure to classify.

[0108] The second artificial intelligence model (520) can determine the priority of the intent of a user utterance based on at least one of user data (542) (e.g., user usage patterns), information of the electronic device (544) (e.g., status of the connected device), and application information (546) (e.g., currently running application). The electronic device (500) can classify the intent of a user utterance by generating a prompt that classifies input text (e.g., user utterance converted into text) into intent units based on time information, user data (542), information of the electronic device (544), and application information (546).

[0109] The second artificial intelligence model (520) can receive a prompt generated based on time information, user data (542), information of the electronic device (544), and application information (546) and determine the intent of the input text. For example, the user may use imperative sentences when making a request to the electronic device (500). The electronic device (500) can determine a relatively high priority for imperative requests among the input texts compared to other texts.

[0110] User data (542) may include, for example, information about services primarily requested by the user or services used by time zone. Information about the electronic device (544) may include information about the on / off status of external electronic devices (e.g., voice assistant, IoT device) registered to the electronic device (500), the operating mode of the external electronic devices, and at least one of the services used by the external electronic devices by time zone. Application information (546) may include information about applications used by the electronic device or applications used by the external electronic devices.

[0111] User data (542) can be stored in a database (e.g., memory (130) of FIG. 1). Information (544) of the electronic device can be received from an external server (505) where an external electronic device registered to a user account is registered. Alternatively, information (544) of the electronic device can be received from a database that is updated in conjunction with an external electronic device (e.g., IoT device) at regular intervals.

[0112] The electronic device (500) can generate a prompt that classifies input text (e.g., converting user utterance into text) into intent units based on time information, user data (542), information of the electronic device (544), and application information (546). The prompt may include a request to divide the input sentence according to meaning and association. The second artificial intelligence model (520) can create multiple sets by making the utterance and meaning into a single set, and can determine the priority of the multiple sets based on time information, user data (542), information of the electronic device (544), and application information (546). The second artificial intelligence model (520) can output the set with the highest priority, and if there are sets with the same priority, it can output all sets with the same priority.

[0113] The electronic device (500) can determine the intent of a user utterance using either the first artificial intelligence model (510) or the second artificial intelligence model (520), and transmit the user utterance and intent to a domain classifier (530). The domain classifier (530) may refer to an LLM with a relatively larger number of parameters than the first artificial intelligence model (510). The second artificial intelligence model (520) can perform domain classification of the user input based on context information (540) when domain classification of the user input fails in the first artificial intelligence model (510). On the other hand, the domain classifier (530) can output a response based on the user's utterance and intent when domain classification of the user input has been performed.

[0114] If the first artificial intelligence model (510) classifies the intent corresponding to the user utterance, the second artificial intelligence model (520) may not be used because the user utterance and intent are transmitted to the domain classifier (530). However, if the first artificial intelligence model (510) fails to classify the intent corresponding to the user utterance, the second artificial intelligence model (520), which has relatively more parameters, can be used to classify the intent corresponding to the user utterance. The number of parameters in an artificial intelligence model can affect the performance of the model. As the artificial intelligence model has more parameters, it can learn more complex patterns, and thus can perform classification of the input text more accurately.

[0115] FIG. 6 illustrates the configuration of a second artificial intelligence model according to one embodiment.

[0116] The second artificial intelligence model (620) (e.g., the second artificial intelligence model (520) of FIG. 5) may include an LLM (622) and a prompt generator (624).

[0117] The second artificial intelligence model (620) can provide the input sentence as input to a large language model (LM) (622). The LLM (622) may refer to a deep learning model that learns text data to understand and generate human language. The LLM (622) can break down the input text into tokens and convert each token into a high-dimensional vector space. Each word in the input sentence can be converted into three vectors: query, key, and value. The LLM (622) can calculate similarity based on the distance between the query vector and all other key vectors. The LLM (622) can determine that the higher the similarity, the more relevant the word is and determine it as the next word.

[0118] For example, the input sentence could be "tell me what I want to eat, weather." The second artificial intelligence model (620) can generate a prompt by providing context information (630) and information about a set priority determination (632) to a prompt generator (624). The second artificial intelligence model (620) can provide the generated prompt and the input sentence to the LLM (622). The context information (630) may include time information, user data (e.g., user data (542) in FIG. 5), information about the electronic device (e.g., information about the electronic device (544) in FIG. 5), and application information (e.g., application information (546) in FIG. 5). The set priority may vary depending on user settings. The electronic device (101) may, for example, determine imperative sentences as having the highest priority. The electronic device (101) can determine, for example, the highest priority for requests related to a specific device (e.g., a TV) (e.g., a request to operate it). Information regarding the priority can be set in advance.

[0119] The prompt may include the following content. The prompt text described below is merely an example, and the content of the prompt is not limited to this.

[0120] [Prompt Text]

[0121] Analyze the input sentences, divide them into semantic units, and summarize them. Different sentences may be entered mixed out of order or in the correct order. Identify the input sentences as having independent meanings based on reference data; if an input sentence has two meanings that are semantically related, group them together and process them as a single sentence. All related sentences, such as those with a precedence relationship, must be returned. Refer to the priority to determine the sentence with the highest priority among the returned sentences, and output a response based on the determined sentence and its semantics. If multiple sentences have equal priority, determine all of them as output types and output a response based on the sentence and its semantics.

[0122] In the prompt text, the reference data may include information about an application running on an electronic device (e.g., the electronic device (101) of FIG. 1). The reference data may include information about an external electronic device currently in operation among the external electronic devices connected to the electronic device (101).

[0123] FIG. 7a illustrates a first embodiment in which an electronic device according to one embodiment outputs a response based on user input.

[0124] The electronic device (700) receives user input and can provide it as input to a first artificial intelligence model (710) (e.g., the first artificial intelligence model (510) of FIG. 5). The first artificial intelligence model (710) may be a model that classifies the utterance intent of the input text. The first artificial intelligence model (710) may have a relatively lower cost compared to the second artificial intelligence model (720) (e.g., the second artificial intelligence model (520) of FIG. 5, the second artificial intelligence model (620) of FIG. 6). Cost may refer to the resources consumed by the artificial intelligence model. The electronic device (700) can classify the intent of the user utterance using the first artificial intelligence model (710) which has a relatively lower cost, and if classification fails, it can benefit in terms of cost by using the second artificial intelligence model (720) which has a relatively higher cost. The electronic device (700) may determine that classification has failed if the probability that the user utterance is included in a specific domain is below a specified level. The electronic device (700) can provide text input to a second artificial intelligence model based on failure to classify.

[0125] In FIG. 7a, the electronic device (700) can classify the utterance intent of the input text using a first artificial intelligence model (710). In this case, the electronic device (700) can transmit the user utterance and intent to a domain classifier (730) (e.g., the domain classifier (530) in FIG. 5) without using a second artificial intelligence model (720). The domain classifier (730) may include an LLM as a model with a relatively larger number of parameters than the first artificial intelligence model (710). The electronic device (700) can generate a prompt instructing the output of a response based on the user utterance and intent. The electronic device (700) can provide the generated prompt as input to the domain classifier (730). The electronic device (700) can output a response to the user input using the domain classifier (730).

[0126] FIG. 7b illustrates a second embodiment in which an electronic device according to one embodiment outputs a response based on user input.

[0127] In FIG. 7b, the electronic device (700) provides user input to the first artificial intelligence model (710) (e.g., the first artificial intelligence model (510) in FIG. 5), but may fail to classify the domain. A domain may refer to a specific topic or field (e.g., medicine, law, sports, finance). For example, in the medical domain, words such as "disease," "symptom," and "treatment" may appear frequently, and in the financial domain, words such as "stock," "investment," and "market" may appear frequently. Based on these characteristics, the LLM can determine, based on probability, which domain the input text belongs to. If the domain is correctly determined, the probability of correctly determining the intent of the user's utterance may increase. The first artificial intelligence model (710) may have difficulty classifying the domain in the case of user input mixed with noise.

[0128] For example, if the user input is "weather, tell me what I want to eat, food," the user input can be classified into "tell me the weather" and "food I want to eat" depending on the intent. Since the first artificial intelligence model (710) has only learned the sentence "tell me the weather," it may decide that the sentence "weather, tell me what I want to eat, food" has not been learned and may not be able to classify the domain. The second artificial intelligence model (720) (e.g., the second artificial intelligence model (520) in FIG. 5, the second artificial intelligence model (620) in FIG. 6) is a model trained with relatively more parameters compared to the first artificial intelligence model, so the domain classification performance may be relatively higher. The second artificial intelligence model (720) can classify the user input into semantic units and divide it into multiple sentences. The second artificial intelligence model (720) can determine the sentence with the highest priority among the multiple classified sentences based on context information (740) and a set priority.

[0129] The second artificial intelligence model (720) can determine the priority of the intent of a user utterance based on at least one of user data (742) (e.g., user usage patterns), information of the electronic device (744) (e.g., status of the connected device), and application information (746) (e.g., currently running application). The electronic device (700) can classify the intent of a user utterance by generating a prompt that classifies input text (e.g., user utterance converted into text) into intent units based on time information, user data (742), information of the electronic device (744), and application information (746).

[0130] The second artificial intelligence model (720) can receive a prompt generated based on time information, user data (742), information of the electronic device (744), and application information (746) and determine the intent of the input text. For example, the user may use imperative sentences when making a request to the electronic device (700). The electronic device (700) can determine a relatively high priority for imperative requests among the input texts compared to other texts.

[0131] According to one embodiment, the second artificial intelligence model (720) can classify the user input into "tell me the weather" and "food I want to eat" according to intent when the user input is "weather, tell me what I want to eat." The second artificial intelligence model (720) can determine that the word "tell me" is more likely to be located after the word "weather" than the word "what I want to eat." The second artificial intelligence model (720) can determine that the word "food" is more likely to be located after the word "what I want to eat" than the word "tell me." Based on these decisions, the second artificial intelligence model (720) can classify the user input into "tell me the weather" and "food I want to eat." The second artificial intelligence model (720) can receive a prompt that determines a relatively high priority for imperative requests among the input texts compared with other texts. The second artificial intelligence model (720) can determine, based on a prompt, that the priority of the imperative request sentence “Tell me the weather” is higher than the priority of the sentence “food I want to eat”. Based on a prompt to output the sentence with the highest priority, the second artificial intelligence model (720) can output only the sentence “Tell me the weather” and pass it to the domain classifier (730). The domain classifier (730) can output a response corresponding to the sentence “Tell me the weather”.

[0132] According to one embodiment, the second artificial intelligence model (720) can classify the user input as "Tell me the weather, what am I eating this morning" according to intent. The second artificial intelligence model (720) checks the user data (742) and determines that the current time is 8:00 AM and that the user inputs utterances regarding the weather more frequently at 8:00 AM. In this case, the second artificial intelligence model (720) can determine that the priority of the sentence "Tell me the weather" is higher than the priority of the sentence "What am I eating this morning?". Additionally, the second artificial intelligence model (720) can receive a prompt that determines a relatively higher priority for imperative requests among the input texts compared to other texts. Based on the prompt, the second artificial intelligence model (720) can determine that the priority of the imperative request sentence "Tell me the weather" is higher than the priority of the sentence "What am I eating this morning?". The second artificial intelligence model (720) can output only the sentence "Tell me the weather" based on a prompt to output the sentence with the highest priority and pass it to the domain classifier (730).

[0133] If the second artificial intelligence model (720) determines that the priority of the sentence “Tell me the weather” and the priority of the sentence “What am I eating for breakfast today?” are the same, it can output both sentences and pass them to the domain classifier (730). If the user input is “Tell me the weather and send it to Mom via text,” the second artificial intelligence model (720) can classify it into “Tell me the weather” and “Send it to Mom via text” according to the intent. The second artificial intelligence model (720) can determine that both sentences are imperative sentences and are linked in a sequential relationship. The second artificial intelligence model (720) can assign equal priority to the two sentences and output both sentences and pass them to the domain classifier (730). The domain classifier (730) can output a response for “Tell me the weather” and a response for “Send it to Mom via text.”

[0134] FIG. 8 illustrates a user interface in which an electronic device according to one embodiment outputs a response based on user input.

[0135] In Fig. 802, an electronic device according to one embodiment (e.g., the electronic device (101) of Fig. 1) can play music and display an interface related to music playback under the control of a processor (e.g., the processor (120) of Fig. 1). In Fig. 804, an electronic device (101) according to one embodiment can receive a user's speech input and convert the input speech into text to display it. The user's speech input may be, for example, "Show me the playlist." A first artificial intelligence model (e.g., the first artificial intelligence model (510) of Fig. 5) can determine that the sentence "Show me the playlist" is a domain related to music and transmit information about the input sentence and the classified domain to a domain classifier (e.g., the domain classifier (530) of Fig. 5). However, since the first artificial intelligence model (510) has not been trained on the noisy sentence "Show me the playlist," domain classification may be difficult. If the electronic device (101) inputs the user's utterance into the first artificial intelligence model (510) but the domain is not classified, the user's utterance can be input into the second artificial intelligence model (820) (e.g., the second artificial intelligence model (520) of FIG. 5, the second artificial intelligence model (620) of FIG. 6). The second artificial intelligence model (820) may refer to an LLM model with relatively more parameters compared to the first artificial intelligence model (510). The second artificial intelligence model (820) can classify the domain of the user's utterance based on information regarding the application running on the electronic device (101). Based on the fact that a music application is currently running, the second artificial intelligence model (820) can determine that the word "list" has the highest probability of appearing after the word "play" in the sentence "show me the play list." The second artificial intelligence model (820) can determine that the word "show me" has the highest probability of appearing after the word "list."The second artificial intelligence model (820) determines that the sentence “Show me the playlist” has the highest priority and can determine the domain as music. The second artificial intelligence model (820) can output information about the sentence “Show me the playlist” and the classified domain and transmit it to the domain classifier (530). The domain classifier (530) can display the music playlist of Figure 806 based on the input sentence and information about the domain.

[0136] FIG. 9 is a flowchart illustrating the operation method of an electronic device according to one embodiment.

[0137] The operations described through FIG. 9 may be implemented based on instructions that can be stored in a computer storage medium or memory (e.g., memory (130) of FIG. 1). The illustrated method may be executed by an electronic device (e.g., electronic device (101) of FIG. 1) described above through FIG. 1 to 8, and the technical features described above will be omitted below. The order of each operation in FIG. 9 may be changed, some operations may be omitted, and some operations may be performed simultaneously. According to one embodiment, operations 910 to 950 may be understood as being performed in a processor of the electronic device (101) (e.g., processor (120) of FIG. 1).

[0138] The memory (130) can store user data related to the user of the electronic device (101) and a plurality of prompt templates.

[0139] In operation 910, an electronic device (101) according to one embodiment may receive user input including a query for an artificial intelligence model under the control of a processor (120). The electronic device (101) may receive user input including a query for an artificial intelligence (AI) model.

[0140] In operation 920, an electronic device (101) according to one embodiment may convert user input into text and input a prompt requesting it to be divided into semantic units under the control of a processor (120). The electronic device (101) may convert user input into text, input the converted text into a first artificial intelligence model (e.g., the first artificial intelligence model (510) of FIG. 5) which has learned information about at least one domain, and input a prompt requesting it to be divided into semantic units.

[0141] In operation 930, an electronic device (101) according to one embodiment can classify input text into semantic units under the control of a processor (120) and determine the probability that the input text is included in a specific domain corresponding to a specific topic among a plurality of domains. The electronic device (101) classifies input text into semantic units and determines the probability that the input text is included in a specific domain corresponding to a specific topic among a plurality of domains, and determines that the input text is included in the specific domain based on the probability of being included in the specific domain exceeding a specified level, and can transmit the converted text and information about the specific domain to a domain classifier.

[0142] In operation 940, an electronic device (101) according to one embodiment may transmit text input to a domain classifier based on the probability of being included in a specific domain, or transmit text input to a second artificial intelligence model (e.g., the second artificial intelligence model (520) of FIG. 5). The electronic device (101) may provide the converted text, the user data, the state of the electronic device (101), and information about the application as input to the second artificial intelligence model (520) based on the fact that the probability of being included in a specific domain corresponding to a specific topic among a plurality of domains as a result of classification by semantic unit is less than a specified level.

[0143] In operation 950, an electronic device (101) according to one embodiment can determine a domain and a user intent and output a response to a user utterance. The electronic device (101) can determine a domain and a user intent based on the output result of a second artificial intelligence model (520) and output a response to the user input.

[0144] According to one embodiment, the first artificial intelligence model (510) may refer to a learning model having a relatively smaller number of parameters compared to the second artificial intelligence model (520). According to one embodiment, in order for a user to obtain a desired response through the AI ​​model, it may be necessary to write a prompt that the AI ​​model can understand well and operate well through the AI ​​model. For example, parameters required for the AI ​​model's response include information such as reader level, response length, respondent's perspective, output format, and output language, and if these parameters are written on the prompt through user input, the AI ​​model can clearly understand the user's query and provide the desired response.

[0145] According to one embodiment, the first artificial intelligence model (510) and the second artificial intelligence model (520) may be placed inside the electronic device (101) and operated by the processor (120), or placed on an external server and operated through a communication connection with the electronic device (101).

[0146] According to one embodiment, the electronic device (101) collects text data for at least one domain and labels it in semantic units, and can train the first artificial intelligence model (510) using the labeled data.

[0147] According to one embodiment, at least one domain may include a personal area including at least one of message transmission and reception, calendar management, alarm setting, and information retrieval; a health management area including at least one of exercise analysis, sleep patterns, heart rate, and stress management; a smart home area including at least one of lighting control, temperature control, IoT device control, and security management; an entertainment area including at least one of music playback, video recommendation, game recommendation, and book recommendation; a work area including at least one of document creation, translation, and material retrieval; and a privacy area including at least one of biometric authentication, data encryption, and personal information management.

[0148] FIG. 10 is a flowchart illustrating the operation method of an electronic device according to one embodiment.

[0149] The operations described through FIG. 10 may be implemented based on instructions that can be stored in a computer storage medium or memory (e.g., memory (130) of FIG. 1). The illustrated method may be executed by an electronic device (e.g., electronic device (101) of FIG. 1) described above through FIG. 1 to 8, and the technical features described above will be omitted below. The order of each operation of FIG. 10 may be changed, some operations may be omitted, and some operations may be performed simultaneously. According to one embodiment, operations 1002 to 1016 may be understood as being performed in a processor of the electronic device (101) (e.g., processor (120) of FIG. 1).

[0150] In operation 1002, an electronic device (101) according to one embodiment can receive a user's utterance under the control of a processor (120).

[0151] In operation 1004, an electronic device (101) according to one embodiment can convert a received utterance into text under the control of a processor (120). The electronic device (101) can receive a user's utterance and convert it into text using a speech recognition technology (e.g., automatic speech recognition (ASR)).

[0152] In operation 1006, an electronic device (101) according to one embodiment can classify text into semantic units using a first artificial intelligence model under the control of a processor (120). The first artificial intelligence model (e.g., the first artificial intelligence model (510) of FIG. 5) may refer to a machine learning model or a large language model (LM) having a relatively small number of parameters compared to the second artificial intelligence model (e.g., the second artificial intelligence model (520) of FIG. 5). The second artificial intelligence model (520) may refer to a large language model (LM) having a relatively large number of parameters compared to the first artificial intelligence model (510).

[0153] In operation 1010, an electronic device (101) according to one embodiment may calculate the probability that input text is included in a specific domain under the control of a processor (120) and determine whether the probability exceeds a specified value. According to one embodiment, the electronic device (101) may use a first artificial intelligence model (510) to represent the converted text as a vector through word embedding, map it to a vector space to extract features representing the subject of the text, and calculate the probability that the converted text belongs to each domain based on the extracted features.

[0154] In operation 1012, an electronic device (101) according to one embodiment may transmit an input text to a second artificial intelligence model (520) based on the fact that the probability of the input text being included in a specific domain under the control of a processor (120) is less than a specified value. The electronic device (101) may classify the converted text into semantic units using the second artificial intelligence model and user data.

[0155] According to one embodiment, the electronic device (101) calculates the probability that the converted text belongs to each domain, classifies the domain with the highest probability as the domain of the converted text, determines that domain classification is difficult if the highest probability among the probabilities that the converted text belongs to each domain is below a specified level, and transmits the converted text to the second artificial intelligence model (520) based on the determination that domain classification is difficult in the first artificial intelligence model (510).

[0156] According to one embodiment, the electronic device (101) transmits the converted text to the second artificial intelligence model (520) based on the determination that the domain classification is difficult in the first artificial intelligence model (510), and can transmit a prompt to the second artificial intelligence model (520) requesting a priority to be determined based on user data, the state of the electronic device (101), and information about the application when the input sentence is analyzed into multiple parts.

[0157] In operation 1014, an electronic device (101) according to one embodiment may transmit text converted to a domain classifier and a classified domain based on the fact that the probability that the input text is included in a specific domain exceeds a specified value under the control of a processor (120). A domain classifier (e.g., a domain classifier (530) of FIG. 5) is a model that classifies which domain the input text belongs to and may mean at least one of a machine learning model, a deep learning model, or a machine learning model.

[0158] In operation 1016, an electronic device (101) according to one embodiment can output a response to a user utterance based on a domain and user intent under the control of a processor (120).

[0159] According to one embodiment, the electronic device (101) can output the sentence with the highest priority in the converted text using a second artificial intelligence model (520), and transmit the output sentence and meaning to a domain classifier to output content that responds to the user's utterance.

[0160] According to one embodiment, the electronic device (101) can classify the intent of a user's utterance by generating a prompt that classifies the input text (e.g., user utterance converted into text) into an intent unit based on time information, user data, information of the electronic device and application information.

[0161] User data may include, for example, information about services primarily requested by the user or services used by time of day. Information about the electronic device may include information about at least one of the on / off status of external electronic devices (e.g., voice assistant, IoT device) registered to the electronic device (101), the operating mode of the external electronic devices, and services used by the external electronic devices by time of day. Application information may include information about applications used by the electronic device or applications used by the external electronic devices.

[0162] According to one embodiment, the prompt may include a request to divide the input sentence according to its meaning and association. The second artificial intelligence model (520) may create multiple sets by making the utterance and meaning into a single set, and may determine the priority of the multiple sets based on time information, user data, information of the electronic device, and application information. The second artificial intelligence model (520) may output the set with the highest priority, and if there are sets with the same priority, it may output all sets with the same priority.

[0163] According to one embodiment, the artificial intelligence model can generate a prompt based on current state information without information on priority.

[0164] According to one embodiment, the electronic device (101) can determine the intent of a user utterance using either a first artificial intelligence model (510) or a second artificial intelligence model (520), and transmit the user utterance and intent to a domain classifier (530). The domain classifier may refer to an LLM with a relatively larger number of parameters than the first artificial intelligence model (510). The second artificial intelligence model (520) can perform domain classification of the user input based on context information when domain classification of the user input fails in the first artificial intelligence model (510). On the other hand, the domain classifier (530) can output a response based on the user's utterance and intent when domain classification of the user input has been performed.

[0165] According to one embodiment, the electronic device (101) can classify sentences into intent units. The electronic device (101) can classify noise within the utterance content through classification into intent units. Even in situations where noise is included in the user's utterance, the electronic device (101) can classify it into meaning units and provide a response that matches the user's intent. Additionally, the electronic device (101) can classify each command and provide an appropriate response for each command in situations where a complex utterance containing multiple commands is recognized.

[0166] Additionally, the electronic device (101) can determine the priority of speech content through the user's usage pattern or information from an external electronic device. The electronic device (101) can distinguish high-priority commands among user inputs by reflecting preferences that vary depending on the user.

[0167] According to one embodiment, the first artificial intelligence model (510) may mean a learning model having a relatively small number of parameters compared to the second artificial intelligence model (520).

[0168] According to one embodiment, the first artificial intelligence model (510) and the second artificial intelligence model (520) may be placed inside an electronic device and operated by a processor, or placed on an external server and operated through a communication connection with the electronic device.

[0169] According to one embodiment, the instructions can be controlled so that, when executed individually or collectively by at least one processor, the electronic device collects text data for at least one domain and labels it in semantic units, and uses the labeled data to train a first artificial intelligence model (510).

[0170] According to one embodiment, at least one domain may include a personal area including at least one of message transmission and reception, calendar management, alarm setting, and information retrieval; a health management area including at least one of exercise analysis, sleep patterns, heart rate, and stress management; a smart home area including at least one of lighting control, temperature control, IoT device control, and security management; an entertainment area including at least one of music playback, video recommendation, game recommendation, and book recommendation; a work area including at least one of document creation, translation, and material retrieval; and a privacy area including at least one of biometric authentication, data encryption, and personal information management.

[0171] According to one embodiment, the instructions can be controlled so that, when executed individually or collectively by at least one processor, the electronic device uses the first artificial intelligence model (510) to represent the converted text as a vector through word embeddings, maps it into a vector space to extract features representing the subject of the text, and calculates the probability that the converted text belongs to each domain based on the extracted features.

[0172] According to one embodiment, instructions can be controlled such that, when executed individually or collectively by at least one processor, the electronic device calculates the probability that the converted text belongs to each domain, classifies the domain with the highest probability as the domain of the converted text, determines that domain classification is difficult if the highest probability among the probabilities that the converted text belongs to each domain is below a specified level, and transmits the converted text to the second artificial intelligence model (520) based on the determination that domain classification is difficult in the first artificial intelligence model (510).

[0173] The electronic device according to the various embodiments disclosed in this document may be of various forms. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a consumer electronics device. The electronic device according to the embodiments of this document is not limited to the devices described above.

[0174] The various embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In this document, phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B and C," and "at least one of A, B, or C" may each include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used simply to distinguish said components from other said components and do not limit said components in any other aspect (e.g., importance or order). Where any (e.g., 1st) component is referred to as “coupled” or “connected” to another (e.g., 2nd) component, with or without the terms “functionally” or “communicationly,” it means that said any component may be connected to said other component directly (e.g., via a wire), wirelessly, or through a third component.

[0175] The term “module” as used in the various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0176] Various embodiments of the present document may be implemented as software (e.g., program (140)) comprising one or more instructions stored in a storage medium (e.g., internal memory (136) or external memory (138)) readable by a machine (e.g., electronic device (101)). For example, a processor (e.g., processor (120)) of the machine (e.g., electronic device (101)) may call at least one of the one or more instructions stored in the storage medium and execute it. This enables the machine to be operated to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by the machine may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily.

[0177] According to one embodiment, the method according to the various embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or distributed online (e.g., download or upload) through an application store (e.g., Play Store™) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0178] According to various embodiments, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to various embodiments, one or more of the components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to integration. According to various embodiments, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0179] The embodiments of this document disclosed in this specification and drawings are merely specific examples presented to facilitate the explanation of the technical content according to the embodiments of this document and to aid in understanding the embodiments of this document, and are not intended to limit the scope of the embodiments of this document. Accordingly, the scope of the embodiments of this document should be interpreted to include all modifications or variations derived based on the technical concept of the embodiments of this document, in addition to the embodiments disclosed herein.

Claims

1. In an electronic device, Memory that stores instructions and includes one or more storage media; It includes at least one processor comprising processing circuitry, and The above memory is, Storing user data related to the user of the electronic device and multiple prompt templates, and When the above instructions are executed individually or collectively by the at least one processor, the electronic device Receive user input including a query for an AI (artificial intelligence) model, and Convert the above user input into text, and Input the converted text into a first artificial intelligence model trained on information for at least one domain, and input a prompt requesting it to be divided into semantic units, and The input text is classified into semantic units to determine the probability of being included in a specific domain corresponding to a specific topic among multiple domains, and the input text is determined to be included in the specific domain based on the probability of being included in the specific domain exceeding a specified level, and the converted text and information about the specific domain are transmitted to a domain classifier. Based on the fact that the probability of being included in a specific domain corresponding to a specific topic among multiple domains as a result of classification by semantic unit is below a specified level, the converted text, the user data, the status of the electronic device, and information about the application are provided as input to a second artificial intelligence model, and An electronic device that determines a domain and user intent based on the output result of the second artificial intelligence model and controls the output of a response to the user input.

2. In Paragraph 1, The above-mentioned first artificial intelligence model is an electronic device that refers to a learning model having a relatively small number of parameters compared to the above-mentioned second artificial intelligence model.

3. In Paragraph 1, The above-mentioned first artificial intelligence model and the above-mentioned second artificial intelligence model Placed inside the electronic device and operated by the processor or An electronic device deployed on an external server and operating through a communication connection with the electronic device.

4. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the electronic device Collect text data for at least one domain and label it by semantic unit, and An electronic device that controls the training of the first artificial intelligence model using labeled data.

5. In Paragraph 1, The above at least one domain is A personal area including at least one of sending and receiving messages, managing calendars, setting alarms, and searching for information, and A health management domain including at least one of exercise analysis, sleep patterns, heart rate, and stress management, A smart home area comprising at least one of lighting control, temperature control, IoT device control and security management, An entertainment area including at least one of music playback, video recommendation, game recommendation and book recommendation, A work area including at least one of document creation, translation, and material retrieval, and An electronic device comprising a privacy area including at least one of biometric authentication, data encryption, and personal information management.

6. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the electronic device Using the above-mentioned first artificial intelligence model, the converted text is represented as a vector through word embedding, and It extracts features representing the topic of the text by mapping them into a vector space, and An electronic device that controls the calculation of the probability that the converted text belongs to each domain based on extracted features.

7. In Paragraph 6, When the above instructions are executed individually or collectively by the at least one processor, the electronic device Calculate the probability that the above converted text belongs to each domain, and The domain with the highest probability is classified as the domain of the above-mentioned transformed text, and If the highest probability among the probabilities of the above-mentioned converted text belonging to each domain is below a specified level, it is determined that domain classification is difficult, and An electronic device that controls the transmission of the converted text to the second artificial intelligence model based on the determination that domain classification is difficult in the first artificial intelligence model.

8. In Paragraph 7, When the above instructions are executed individually or collectively by the at least one processor, the electronic device Based on the determination that domain classification is difficult in the first artificial intelligence model, the converted text is transmitted to the second artificial intelligence model, and An electronic device that controls the transmission of a prompt to the second artificial intelligence model requesting a priority to be determined based on the user data, the status of the electronic device, and information about the application when an input sentence is analyzed into multiple parts.

9. In Paragraph 8, When the above instructions are executed individually or collectively by the at least one processor, the electronic device Using the above second artificial intelligence model, output the sentence with the highest priority in the above converted text, and An electronic device that transmits the output sentence and meaning to the domain classifier to control the output of content responding to the user's utterance.

10. In Paragraph 1, The above-mentioned first artificial intelligence model It refers to a machine learning model or LLM (large language model) with a relatively small number of parameters compared to the second artificial intelligence model mentioned above, and The above second artificial intelligence model It refers to a large language model (LM) with a relatively large number of parameters compared to the first artificial intelligence model mentioned above, and The above domain classifier is An electronic device that refers to at least one of a machine learning model or a deep learning model, which classifies which domain an input text belongs to.

11. A computer-readable non-transient storage medium storing one or more programs comprising instructions executable by at least one processor of an electronic device, Receive user input including a query for an AI (artificial intelligence) model, and Convert the above user input into text, and Input the converted text into a first artificial intelligence model trained on information for at least one domain, and input a prompt requesting it to be divided into semantic units, and The input text is classified into semantic units to determine the probability of being included in a specific domain corresponding to a specific topic among multiple domains, and the input text is determined to be included in the specific domain based on the probability of being included in the specific domain exceeding a specified level, and the converted text and information about the specific domain are transmitted to a domain classifier. Based on the fact that the probability of being included in a specific domain corresponding to a specific topic among multiple domains as a result of classification by semantic unit is below a specified level, the converted text and user data, and information about the status of the electronic device and the application are provided as input to a second artificial intelligence model, A computer-readable non-transient storage medium that controls the output of a response to a user input by determining a domain and user intent based on the output result of the second artificial intelligence model.

12. In Paragraph 11, A computer-readable non-transient storage medium in which the first artificial intelligence model above refers to a learning model having a relatively small number of parameters compared to the second artificial intelligence model above.

13. In Paragraph 11, The above-mentioned first artificial intelligence model and the above-mentioned second artificial intelligence model Placed inside the electronic device and operated by the processor or A computer-readable non-transient storage medium that is placed on an external server and operates through a communication connection with the electronic device.

14. In Paragraph 11, When the above instructions are executed individually or collectively by the at least one processor, the electronic device Collect text data for at least one domain and label it by semantic unit, and A computer-readable non-transient storage medium for controlling the training of the first artificial intelligence model using labeled data.

15. In Paragraph 11, The above at least one domain is A personal area including at least one of sending and receiving messages, managing calendars, setting alarms, and searching for information, and A health management domain including at least one of exercise analysis, sleep patterns, heart rate, and stress management, A smart home area comprising at least one of lighting control, temperature control, IoT device control and security management, An entertainment area including at least one of music playback, video recommendation, game recommendation and book recommendation, A work area including at least one of document creation, translation, and material retrieval, and A computer-readable non-transient storage medium comprising a privacy area including at least one of biometric authentication, data encryption, and personal information management.

Images