Electronic device, method, and non-transitory computer-readable storage medium for providing at least one parameter to server

Abstract

A method performed by an electronic device may comprise the operations of: receiving a language-based request from a user of the electronic device; transmitting, via the communication circuit, first data pertaining to the language-based request to a server including a trained model; receiving, from the server via the communication circuit, second data, which is generated on the basis of the trained model and corresponds to the first data, in order to provide a response corresponding to the language-based request; acquiring third data, which is stored in the electronic device and related to the language-based request, according to the second data on the basis of data acquired through a plurality of applications by the electronic device; transmitting, to the server, information pertaining to the third data stored in the electronic device in order to acquire output data on the basis of the third data stored in the electronic device; receiving the output data from the server; and providing a response to the language-based request by using the output data.

Description

Electronic device, method, and non-transitory computer-readable storage medium for providing at least one parameter to a server

[0001] The following descriptions relate to electronic devices, methods, and non-transitory computer-readable storage media for providing at least one parameter to a server.

[0002] Electronic devices can provide services that perform functions at the user's request using interactive applications. The electronic devices can identify the user's voice input and perform functions based on the voice input. Using an artificial intelligence model, the electronic devices can identify the function requested by the user based on the voice input. The electronic devices can then perform the function requested by the user.

[0003] The above information may be provided as background art to aid in understanding the present disclosure. No claim or determination is made as to whether any of the above-described matters constitute prior art related to the present disclosure.

[0004] According to one embodiment, an electronic device may include at least one processor including communication circuitry, a memory including instructions and one or more storage media, and a processing circuit. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to receive a language-based request from a user of the electronic device, transmit first data regarding the language-based request to a server including a trained model through the communication circuit, receive second data generated based on the trained model and corresponding to the first data from the server through the communication circuit to provide a response corresponding to the language-based request, obtain third data related to the language-based request based on data acquired through a plurality of applications on the electronic device, and transmit information regarding the third data stored in the electronic device to the server to obtain output data based on the third data stored in the electronic device, receive the output data from the server, and provide a response to the language-based request using the output data.

[0005] According to one embodiment, a method performed by an electronic device may include receiving a language-based request from a user of the electronic device, transmitting, through the communication circuit, first data regarding the language-based request to a server including a trained model, receiving, through the communication circuit, from the server, second data generated based on the trained model and corresponding to the first data to provide a response corresponding to the language-based request, obtaining, based on data acquired through a plurality of applications of the electronic device, third data stored in the electronic device and related to the language-based request according to the second data, transmitting information regarding the third data stored in the electronic device to the server to obtain output data based on the third data stored in the electronic device, receiving, from the server, the output data, and providing a response to the language-based request using the output data.

[0006] According to one embodiment, a non-transitory computer-readable storage medium can store one or more programs. The one or more programs may include instructions that, when executed by at least one processor of an electronic device including a communication circuit, cause the electronic device to receive a language-based request from a user of the electronic device, transmit first data regarding the language-based request to a server including a trained model through the communication circuit, receive second data generated based on the trained model and corresponding to the first data from the server through the communication circuit to provide a response corresponding to the language-based request, obtain third data related to the language-based request based on data acquired through a plurality of applications of the electronic device, and transmit information regarding the third data stored in the electronic device to the server to obtain output data based on the third data stored in the electronic device, receive the output data from the server, and use the output data to cause the electronic device to provide a response to the language-based request.

[0007] According to one embodiment, an electronic device may include a communication circuit, a memory including instructions and one or more storage media, and at least one processor including a processing circuit. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to receive a language-based request from a user of the electronic device, transmit first data regarding the language-based request to a server including a trained model through the communication circuit, receive first output data generated based on the trained model and second data corresponding to the first data from the server through the communication circuit to provide a response corresponding to the language-based request, obtain third data stored in the electronic device based on the second data based on data obtained through a plurality of applications of the electronic device, obtain second output data based on the third data and the first output data, and provide a response to the language-based request using the second output data.

[0008] FIG. 1 is a block diagram of an electronic device within a network environment according to one embodiment.

[0009] Figures 2a and 2b illustrate examples of operation of an interactive application according to one embodiment.

[0010] FIG. 2c illustrates an example of a simplified block diagram of an electronic device, according to one embodiment.

[0011] FIG. 3A illustrates a flowchart of the operation of an electronic device according to one embodiment.

[0012] FIG. 3b illustrates a flowchart of the operation of an electronic device according to one embodiment.

[0013] FIG. 4 illustrates examples of functional blocks of an electronic device and a server according to one embodiment.

[0014] FIG. 5 illustrates an example of the operation of an electronic device and a server according to one embodiment.

[0015] Figure 6 illustrates an example of the operation of a server according to one embodiment.

[0016] FIG. 7 illustrates an example of operation of an electronic device according to one embodiment.

[0017] FIG. 8 illustrates an example of operation of an electronic device according to one embodiment.

[0018] FIGS. 9A and 9B illustrate examples of operation of an electronic device according to one embodiment.

[0019] FIGS. 10A and 10B illustrate examples of operation of an electronic device according to one embodiment.

[0020] FIGS. 11A, 11B, and 11C illustrate examples of operation of an electronic device according to one embodiment.

[0021] Figure 12 illustrates an example of the operation of an electronic device according to one embodiment.

[0022] Figure 13 is a block diagram illustrating an integrated intelligence system according to one embodiment.

[0023] FIG. 14 is a diagram showing a form in which relationship information between concepts and actions is stored in a database according to various embodiments.

[0024] FIG. 15 is a diagram showing a screen for processing voice input received through an intelligent app by a user terminal according to various embodiments.

[0025] Figure 16 is a schematic diagram of an exemplary AI system.

[0026] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. In connection with the description of the drawings, the same or similar reference numerals may be used for identical or similar components. Furthermore, in the drawings and related descriptions, descriptions of well-known functions and configurations may be omitted for clarity and conciseness.

[0027] FIG. 1 is a block diagram of an electronic device within a network environment according to one embodiment.

[0028] Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of an electronic device (104) or a server (108) via a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0029] The processor (120) may control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) by executing, for example, software (e.g., a program (140)), and may perform various data processing or calculations. According to one embodiment, as at least a part of the data processing or calculation, the processor (120) may store a command or data received from another component (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the command or data stored in the volatile memory (132), and store the resulting data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or a secondary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together therewith. For example, if the electronic device (101) includes a main processor (121) and a secondary processor (123), the secondary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a specified function. The secondary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0030] The auxiliary processor (123) may control at least a part of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0031] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0032] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0033] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0034] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. According to one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0035] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0036] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0037] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0038] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0039] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0040] A haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. According to one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0041] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0042] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0043] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0044] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0045] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) may support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0046] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). According to one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas, for example, by the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0047] According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

[0048] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0049] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service by itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0050] According to one embodiment, an electronic device (e.g., electronic device (101)) may provide a conversational artificial intelligence service using a conversational application. The electronic device may receive a language-based request. For example, the language-based request may be composed based on at least one of text and voice.

[0051] An electronic device can input input data containing language-based requests into a trained model (e.g., an artificial intelligence model). The electronic device can obtain output data based on the output of the trained model. The electronic device can display a language-based response message based on the output data through the user interface of an interactive application or perform a function based on the output data.

[0052] According to one embodiment, a trained model for obtaining output data based on a language-based request may be included in a server. An electronic device may transmit a language-based request to the server. The server may obtain output data based on inputting the language-based request to the trained model. The server may transmit the output data to the electronic device. The electronic device may display a language-based response message based on the output data through a user interface of an interactive application or perform a function based on the output data.

[0053] However, the server may not be able to access information stored in the electronic device. Therefore, the server may not be able to identify parameters for obtaining output data. The server may request parameters for obtaining output data from the electronic device. The electronic device may search for resources available within the electronic device and obtain a plurality of candidate parameters. The electronic device may identify at least one of the candidate parameters as at least one parameter to be provided to the server. The electronic device may receive output data from the server based on transmitting at least one parameter to the server. The electronic device may use the output data to provide a response to a language-based request.

[0054] In the following specification, the operation of an electronic device for obtaining a plurality of candidate parameters and identifying at least one parameter among the plurality of candidate parameters will be described.

[0055] Figures 2a and 2b illustrate examples of operation of an interactive application according to one embodiment.

[0056] Referring to FIGS. 2A and 2B, the electronic device (200) may include the electronic device (101) of FIG. 1. The electronic device (200) may be a terminal owned by a user. The terminal may include, for example, a personal computer (PC) such as a laptop or desktop, a smartphone, a smartpad, or a tablet PC. The terminal may include a smart accessory such as a smartwatch and / or a head-mounted device (HMD).

[0057] According to one embodiment, the electronic device (200) can execute an interactive application. For example, the electronic device (200) can execute an interactive application based on a defined utterance. The electronic device (200) can identify an utterance based on a user's voice signal. The electronic device (200) can execute an interactive application based on determining whether the identified utterance corresponds to the defined utterance. For example, the interactive application may be referred to as an artificial intelligence assistant application.

[0058] According to one embodiment, an interactive application may be used to provide various functions using a first trained model (e.g., an artificial intelligence model). For example, the first trained model may be contained in a server. The electronic device (200) may transmit input data for the first trained model to the server. The server may provide output data for the first trained model to the electronic device (200).

[0059] Referring to FIG. 2A, the electronic device (200) may display a user interface (290) of the interactive application based on the execution of the interactive application. The electronic device (200) may receive a language-based request within the user interface (290). The electronic device (200) may display an object (206) representing the received language-based request on the user interface (290).

[0060] According to one embodiment, the electronic device (200) may transmit input data including a language-based request to a server. The electronic device (200) may receive information from the server including an intent included in the language-based request. For example, the server may identify that additional parameters are required to obtain output data. Based on transmitting information including the intent included in the language-based request to the electronic device (200), the server may request the electronic device (200) to transmit parameters for obtaining the output data.

[0061] According to one embodiment, the electronic device (200) may search available resources within the electronic device (200) to obtain parameters related to an intent within information received from a server. Within resources (209) related to a messenger application, parameters related to the intent may be obtained. For example, parameters related to the intent may include words (208) indicating a location, such as "Gangnam A Restaurant."

[0062] Although not illustrated, the electronic device (200) may obtain a plurality of candidate parameters within a resource related to a text application, a resource related to a schedule (or calendar) application, and a resource related to a memo application. The electronic device (200) may identify at least one candidate parameter among the plurality of candidate parameters using the second trained model. For example, the electronic device (200) may identify a candidate parameter obtained within a resource related to a messenger application (209) among the candidate parameters as a parameter to be provided to the server.

[0063] According to one embodiment, the electronic device (200) can transmit parameters related to an intent to a server. The server can obtain output data using a first trained model based on the received parameters and input data. The server can transmit the output data to the electronic device (200). Based on the output data received from the server, the electronic device (200) can display an object (207) on a user interface (290). Based on the display of the object (207), the electronic device (200) can provide a response to a language-based request.

[0064] According to one embodiment, as described above, the electronic device (200) may transmit to the server at least one parameter necessary for obtaining output data from the server without transmitting all of the resources (or information) contained in the electronic device (200). The electronic device (200) may provide to the server only at least one parameter necessary for obtaining output data without transmitting to the server character information, schedule information, and / or memo information corresponding to personal information.

[0065] According to one embodiment, the electronic device (200) may provide (or transmit) data generated based on input data as well as at least one parameter to the server. For example, the electronic device (200) may perform inference and / or learning about a situation based on the input data. The electronic device (200) may generate data based on the inference and / or learning, and provide (or transmit) the generated data to the server.

[0066] Referring to FIG. 2B, the electronic device (200) may display a user interface (290) of the interactive application based on the execution of the interactive application. The electronic device (200) may receive a language-based request within the user interface (290). The electronic device (200) may display an object (215) representing the received language-based request on the user interface (290).

[0067] According to one embodiment, the electronic device (200) may transmit input data including a language-based request to a server. The electronic device (200) may receive information including an intent included in the language-based request from the server. For example, the server may identify that additional parameters are required to obtain output data. Based on transmitting the information including the intent included in the language-based request to the electronic device (200), the server may request that the electronic device (200) transmit parameters for obtaining the output data. According to one embodiment, the server may transmit the input data and at least one parameter stored in the server, and request that the parameters for obtaining the output data be transmitted. For example, based on identifying that additional parameters are required through the input data, the server may identify additional parameters through information (e.g., personalized information) stored in the electronic device (200), and transmit the input data and at least one parameter stored in the server, and request that the parameters for obtaining the output data be transmitted.

[0068] According to one embodiment, the electronic device (200) may search available resources within the electronic device (200) to obtain parameters related to an intent within information received from a server. Within a resource (280) relating to a calendar application, parameters related to the intent may be obtained. For example, parameters related to the intent may include words indicating a schedule, such as "part-weekly meeting." For example, parameters related to the intent may include "part-weekly meeting 2 PM," a schedule for day 3 within the resource (280), and "part-weekly meeting 4 PM," a schedule for day 10. For example, parameters related to the intent may include "Friday 2 PM" and "Friday 4 PM." The parameters related to the intent may be set in various ways depending on the embodiment.

[0069] The electronic device (200) can transmit parameters related to the intent to the server. The server can obtain output data using the first trained model based on the received parameters and input data. The server can transmit the output data to the electronic device (200). The electronic device (200) can display an object (216) on the user interface (290) based on the output data received from the server. The electronic device (200) can provide candidate responses as a response to the language-based request based on the output data. The electronic device (200) can display objects (218) representing the candidate responses on the user interface (290).

[0070] For example, the electronic device (200) may transmit parameters including "2 o'clock" and "4 o'clock" to the server in relation to a weekly meeting. Instead of prompting the user with a question such as "What time would you like to schedule the weekly meeting for?", the server may provide candidate responses, such as 2 o'clock or 4 o'clock, based on the history identified by the parameters. In an embodiment, if the electronic device (200) transmits a parameter including "4 o'clock" in relation to the weekly meeting to the server, the electronic device (200) may provide a response indicating that the weekly meeting this week will be set for 4 o'clock based on the output data.

[0071] According to FIGS. 2A and 2B , when transmitting parameters required for a response to a language-based request to a server, the electronic device (200) may select (or filter) at least one parameter from among a plurality of parameters acquired within the electronic device (200) using the second trained model. The electronic device (200) may transmit at least one parameter for acquiring output data to the server through the first trained model. For example, the electronic device (200) may provide the server with parameters (or information) that are likely to be used for acquiring output data through the first trained model.

[0072] Components of the electronic device (200) according to the above-described embodiments will be described later in FIG. 2c.

[0073] FIG. 2c illustrates an example of a simplified block diagram of an electronic device, according to one embodiment.

[0074] Referring to FIG. 2c, the electronic device (200) may include at least some or all of the components of the electronic device (101) of FIG. 1. For example, the electronic device (200) may correspond to the electronic device (101) of FIG. 1.

[0075] According to one embodiment, the electronic device (200) may include at least one of a processor (201), a display (202), a memory (203), and / or a communication circuit (204). For example, at least some of the processor (201), the display (202), the memory (203), and / or the communication circuit (204) may be omitted depending on the embodiment.

[0076] According to one embodiment, the processor (201) may include at least a portion of the processor (120) of FIG. 1 or may correspond to at least a portion of the processor (120). For example, the processor (201) may include one or more processors, including an application processor (AP) and / or a communication processor (CP). For example, the processor (201) may be implemented as a single chip, such as a system on chip (SoC), or may be implemented as multiple chips. For example, the processor (201) may be implemented as a single integrated circuit or may be implemented as multiple integrated circuits. For example, the processor (201) may be distributedly arranged within the electronic device (200).

[0077] According to one embodiment, the processor (201) may be operatively or operably coupled with or connected with the display (202), the memory (203), and the communication circuitry (204). For example, operatively coupling the processor (201) with another component may mean that the processor (201) can control the other component. The processor (201) may control the display (202), the memory (203), and / or the communication circuitry (204).

[0078] According to one embodiment, the display (202) of the electronic device (200) can output visualized information (e.g., a screen) to the user. For example, the display (202) can be controlled by a controller, such as a graphic processing unit (GPU), to output visualized information to the user. The display (202) can include a liquid crystal display (LCD), a plasma display panel (PDP), and / or one or more light emitting diodes (LEDs). The LEDs can include organic LEDs (OLEDs). The display (202) can include a flat panel display (FPD) and / or electronic paper. The embodiment is not limited thereto, and the display (202) can have an at least partially curved shape or a deformable shape. A display (202) having a deformable shape can be referred to as a flexible display.

[0079] According to one embodiment, the memory (203) of the electronic device (200) may include a circuit and / or a storage medium for storing data and / or instructions input and / or output to the processor (201). The memory (203) may include, for example, volatile memory such as random-access memory (RAM) and / or non-volatile memory such as read-only memory (ROM). The non-volatile memory may be referred to as storage. The volatile memory may include, for example, at least one of dynamic RAM (DRAM), static RAM (SRAM), cache RAM, and pseudo SRAM (PSRAM). The non-volatile memory may include, for example, at least one of programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), flash memory, hard disk, compact disc, solid state drive (SSD), and embedded multi media card (eMMC).

[0080] According to one embodiment, the memory (203) may include at least a portion of the memory (130) of FIG. 1 or may correspond to at least a portion of the memory (130) of FIG. 1. For example, the memory (203) may be implemented as a single chip or as multiple chips. For example, the memory (203) may be implemented as a single integrated circuit or as multiple integrated circuits. For example, the memory (203) may be distributedly arranged within the electronic device (200).

[0081] According to one embodiment, the processor (201) of the electronic device (200) may execute instructions of the memory (203) within the electronic device (200) to perform functions and / or operations indicated by the instructions. For example, when the electronic device (200) includes at least one processor, the at least one processor may be configured to collectively or individually execute the instructions.

[0082] For example, the memory (203) may include at least one model (or at least one artificial intelligence model). The memory (203) may store instructions relating to the at least one model. The memory (203) may include a second trained model, which will be described below. In some embodiments, the memory (203) may also include a first trained model.

[0083] For example, the second trained model may be constructed based on an artificial intelligence model. Depending on the embodiment, the second trained model may be constructed based on at least one of a rule-based model, a pattern-based model, and / or a deep model. The second trained model may be constructed based on a generative model (or a generative artificial intelligence model). However, the present invention is not limited thereto.

[0084] In one embodiment, the generative model may include a generative model including a plurality of parameters associated with a neural network having a structure based on an encoder and a decoder, such as a transformer. In one embodiment, the generative model may include a bidirectional model based on learning about an encoder (e.g., bidirectional encoder representations from transformers (BERT)) or an auto-encoding model (e.g., a diffusion model). In one embodiment, the generative model may include an auto-regressor model based on learning about a decoder (e.g., a generative pre-trained transformer (GPT)). In one embodiment, the generative model may include a sequence-to-sequence model based on learning about an encoder and a decoder (e.g., stable diffusion, DALL-E 2). In one embodiment, the generative model may include, but is not limited to, a large language model (LLM) for processing natural language based on a massive number of parameters. The generative model may include parameters for driving a neural network such as a convolutional neural network (CNN), a recurrent neural network (RNN), a feedforward neural network (FNN), and / or a long short-term memory (LSTM).

[0085] According to one embodiment, the communication circuit (204) can be used for various radio access technologies (RATs). For example, the communication circuit (204) can be used to perform Bluetooth communication, wireless local area network (WLAN) communication, or ultra wideband (UWB) communication. For example, the communication circuit (204) can be used to perform cellular communication. For example, the processor (201) can establish a connection with an external electronic device (e.g., a server) through the communication circuit (204).

[0086] Although not illustrated, a server (e.g., server (300)) described below may also include at least some or all of the components (e.g., processor (201), display (202), memory (203), and / or communication circuit (204)) of the electronic device (200) described above.

[0087] Figure 3a illustrates a flowchart of the operation of an electronic device according to one embodiment. In the following embodiments, the operations may be performed sequentially, but are not necessarily sequential. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0088] Referring to FIG. 3A, at operation 310, the electronic device (200) (or the processor (201) of the electronic device (200)) may receive a language-based request from a user of the electronic device (200).

[0089] According to one embodiment, the electronic device (200) may display a user interface of the interactive application through the display (202) based on the execution of the interactive application. While the user interface of the interactive application is displayed, the electronic device (200) may obtain a language-based request. For example, the language-based request may include at least one of a text input and / or a voice input. For example, the language-based request may include multimedia data including at least one image and / or at least one video.

[0090] In operation 320, the electronic device (200) may transmit input data including at least a portion of a language-based request to a server including a first trained model (e.g., a generative model, LLM). For example, the server may include a first trained model used to provide a response to the language-based request. The electronic device (200) may transmit input data for the first trained model to the server. The electronic device (200) may transmit input data including at least a portion of the language-based request to the server to provide a response to the language-based request. For example, the input data including at least a portion of the language-based request may be referred to as first data regarding the language-based request.

[0091] In operation 330, the electronic device (200) may receive information from the server that includes an intent included in the language-based request. For example, the electronic device (200) may receive second data generated based on the first trained model and corresponding to the first data to provide a response corresponding to the language-based request. As an example, the second data may include information that includes an intent included in the language-based request.

[0092] In one embodiment, the server may input input data into a first trained model. The server may identify that additional parameters are required to obtain output data from the first trained model. To request the additional parameters from the electronic device (200), the server may transmit information including an intent contained within a language-based request to the electronic device (200).

[0093] For example, a language-based request may include an intent. The server may identify the intent contained within the language-based request based on input data that includes at least a portion of the language-based request. The electronic device (200) may transmit information including the identified intent to the server.

[0094] For example, information including an intent may include information regarding the type of parameter (or data) used for output data. For example, the type of parameter (or data) may include date, time, location, and / or target. The electronic device (200) may identify the type of parameter to be searched (or the search target) based on the information including the intent.

[0095] In one embodiment, the server may transmit input data and at least one parameter stored in the server to the electronic device (200), and request the transmission of parameters for obtaining output data. For example, the server may identify that additional parameters (or data) are needed through the input data, and may request the transmission of the input data and at least one parameter stored in the server, and request the transmission of parameters for obtaining output data so that the server can identify additional parameters (or data) through information (e.g., personalized information) stored in the electronic device (200).

[0096] In operation 340, the electronic device (200) may search for resources available within the electronic device (200). For example, the electronic device (200) may search for resources available within the electronic device (200) to obtain parameters related to an intent.

[0097] According to one embodiment, the resources available within the electronic device (200) may include resources acquired based on at least one software included (or stored) in the electronic device (200) or resources acquired based on at least one hardware included in the electronic device (200). For example, the resources available within the electronic device (200) may be referenced as data acquired through multiple applications.

[0098] According to one embodiment, the application for obtaining data can be designated (or changed) by the user. For example, data can be obtained from applications other than applications containing personal information (or privacy information) (e.g., a diary application, a wallet application, or a financial application). For example, the electronic device (200) can identify an application for retrieving data. The electronic device (200) can retrieve data obtained through the identified application. For example, the electronic device (200) can retrieve data obtained through at least one application other than the designated application (e.g., a financial application, a women's health application, a health application, or a security application).

[0099] For example, resources available within the electronic device (200) may include data acquired through applications stored within the electronic device (200). As an example, resources available within the electronic device (200) may include at least one of schedule information stored through a calendar application, conversation history information stored through a messenger application, contact information stored through a contact application, and / or visit history information stored through an Internet application.

[0100] For example, the resources available within the electronic device (200) may include data acquired through at least one sensor (e.g., a light sensor). For example, the resources available within the electronic device (200) may include data acquired through a communication circuit (e.g., a GNSS communication module).

[0101] According to one embodiment, the electronic device (200) may obtain a plurality of candidate parameters based on searching for resources available within the electronic device (200). For example, the electronic device (200) may obtain a plurality of candidate parameters related to an intent included in a language-based request. For example, the operation of obtaining a plurality of candidate parameters may be referred to as the operation of obtaining candidate data.

[0102] For example, the electronic device (200) can identify at least one query to obtain parameters related to an intent. For example, resources available within the electronic device (200) can be configured based on various formats. The electronic device (200) can configure at least one query to obtain parameters related to an intent within the resources configured based on various formats. The electronic device (200) can obtain a plurality of candidate parameters from the resources available within the electronic device (200) based on the at least one query.

[0103] For example, the electronic device (200) can set a search range for obtaining parameters related to an intent. The electronic device (200) can search for resources available within the electronic device (200) within the set search range. For example, the electronic device (200) can set the search range based on at least one of time, location, date, service, parameter type, and / or resource format. For example, the electronic device (200) can perform a search within a specified date range. For example, the electronic device (200) can perform a search for parameters related to a reference distance range from a specified location. For example, the search range can be set based on the policy or logic (or business logic) of an application, a service, and / or a sensor.

[0104] In operation 350, the electronic device (200) may use a second trained model (e.g., a generative model, LLM) to identify at least one candidate parameter among a plurality of candidate parameters as at least one parameter to be provided to the server. For example, the electronic device (200) may use a second trained model within the electronic device (200) that has a lower complexity than the first trained model to identify at least one candidate parameter among a plurality of candidate parameters obtained through the search as at least one parameter to be provided to the server.

[0105] For example, if no candidate parameters are identified, the electronic device (200) may generate a prompt to request information from the user to identify at least one parameter. Based on the information obtained based on the prompt, the electronic device (200) may identify at least one parameter.

[0106] For example, if a candidate parameter is identified as one, the electronic device (200) may perform an action based on the candidate parameter or may inquire the user as to whether the candidate parameter has been properly identified. If the candidate parameter has been properly identified, the electronic device (200) may identify the candidate parameter as at least one parameter to be provided to the server.

[0107] For example, if two or more candidate parameters are identified, the electronic device (200) may inquire about an appropriate candidate parameter among the two or more candidate parameters. The electronic device may identify the candidate parameter determined based on the user's response as at least one parameter to be provided to the server.

[0108] According to one embodiment, the electronic device (200) can input a plurality of parameters into the second trained model. Based on the output of the second trained model, the electronic device (200) can obtain (or identify) at least one candidate parameter.

[0109] For example, the second trained model may not be able to process input data exceeding a reference size. The electronic device (200) may divide a plurality of candidate parameters into one or more parameter sets based on a reference size that can be processed by the second trained model. For example, the reference size may be referred to as a batch size. The electronic device (200) may sequentially input each of the one or more parameter sets into the second trained model. The electronic device (200) may identify at least one candidate parameter based on the outputs of the second trained model.

[0110] According to one embodiment, the complexity of the second trained model included in the electronic device (200) may be lower than the complexity of the first trained model. According to one embodiment, the size of the second trained model may be smaller than the size of the first trained model. According to one embodiment, the number of parameters of the first trained model may be greater than the number of parameters of the second trained model. According to one embodiment, the number of layers of the first trained model may be greater than the number of layers of the second trained model. According to one embodiment, the number of nodes of the first trained model may be greater than the number of nodes of the second trained model.

[0111] According to one embodiment, the electronic device (200) may acquire third data stored in the electronic device based on second data obtained through multiple applications of the electronic device (200). For example, the third data may include at least one candidate parameter.

[0112] In operation 360, the electronic device (200) may transmit data including at least one parameter to the server. For example, the electronic device (200) may transmit data related to an intent to the server. As an example, the data related to the intent may include at least one parameter.

[0113] According to one embodiment, the electronic device (200) may remove parameters related to the user's restricted personal information from at least one candidate parameter. For example, the at least one candidate parameter may include parameters related to the user's restricted personal information. To prevent the parameters related to the user's restricted personal information from being transmitted to the server, the electronic device (200) may remove parameters related to the user's restricted personal information from the at least one candidate parameter. For example, the user's restricted personal information may include contact information, credentials, passwords, account information, and / or card information. For example, the electronic device (200) may perform encryption on the at least one candidate parameter. The electronic device (200) may encrypt the at least one candidate parameter and transmit it to the server.

[0114] For example, the electronic device (200) may provide an interface for removing parameters related to the user's restricted personal information from at least one candidate parameter. The electronic device (200) may provide the user with the ability to select (or change) parameters related to personal information from among the at least one candidate parameter.

[0115] According to one embodiment, the electronic device (200) may provide a method for selecting (or changing) personal information data among the data to be transmitted to the server. The selected data may not be transmitted to the server. For example, the personal information data may include the user's financial data, password data, data obtained in a secret mode of an application (e.g., an Internet application), and / or data stored in a secure area (or secure application). Depending on the embodiment, the personal information data may be changed by the user.

[0116] According to one embodiment, the electronic device (200) may determine data to be transmitted to the server based on the type of data. For example, the electronic device (200) may not transmit data related to a security application, data stored in a secure area, data related to finance, data related to passwords, or data requiring security to the server. The electronic device (200) may remove data related to a security application, data stored in a secure area, data related to finance, data related to passwords, or data requiring security from among the acquired data. For example, the electronic device (200) may not transmit data acquired through a designated application to the server. For example, the electronic device (200) may not transmit data acquired through a designated application (e.g., a financial application, a women's health application, a health application, a security application) to the server. The electronic device (200) may transmit data excluding data acquired through a designated application to the server. Depending on the embodiment, the designated application may vary depending on the user.

[0117] In operation 370, the electronic device (200) may receive output data from a server. For example, the output data may be generated based on a first trained model into which data including input data and at least one parameter has been input. For example, the input data and at least one parameter may be set as input data of the first trained model. The output data may be obtained by inputting the input data and at least one parameter into the first trained model.

[0118] In operation 380, the electronic device (200) may provide a response to the language-based request. For example, the electronic device (200) may provide a response to the language-based request using output data.

[0119] For example, a response to a language-based request may include performing a function related to the application, executing the application, and / or answering a question.

[0120] Figure 3b illustrates a flowchart of the operation of an electronic device according to one embodiment. In the following embodiments, the operations may be performed sequentially, but are not necessarily sequential. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0121] Referring to FIG. 3B, at operation 391, the electronic device (200) (or the processor (201) of the electronic device (200)) may receive a language-based request from a user of the electronic device (200). Operation 391 may correspond to operation 310 of FIG. 3A.

[0122] At operation 392, the electronic device (200) may transmit input data including at least a portion of a language-based request to a server including a first trained model (e.g., a generative model, LLM). Operation 392 may correspond to operation 320 of FIG. 3A.

[0123] According to one embodiment, the electronic device (200) may transmit first data regarding a language-based request to a server including a first trained model (e.g., a generative model, LLM). For example, the first data may include at least a portion of the language-based request.

[0124] In operation 393, the electronic device (200) may receive information from the server, including first output data and an intent included in a language-based request. For example, the first output data may lack some information to provide a response to the language-based request.

[0125] According to one embodiment, a server may obtain first output data using a first trained model. The server may input input data into the first trained model. The server may identify that additional parameters are required to obtain the output data of the first trained model. The server may obtain the first output data without the additional parameters. The server may transmit information including the first output data and an intent to the electronic device (200). The information including the intent may be used to cause the electronic device (200) to obtain the additional parameters. For example, the information including the intent may include information indicating the type of data associated with the intent used to obtain the second output data.

[0126] In operation 394, the electronic device (200) may obtain first output data and data related to the intent. For example, the electronic device (200) may obtain data related to the intent based on data obtained through multiple applications of the electronic device (200). The electronic device (200) may obtain data related to the intent from among the data obtained through multiple applications.

[0127] According to one embodiment, the electronic device (200) may receive first output data generated based on the first trained model and second data corresponding to the first data to provide a response corresponding to a language-based request from the server. For example, the second data may include data related to an intent.

[0128] According to one embodiment, the electronic device (200) can acquire data related to an intent based on searching data acquired through multiple applications. For example, the electronic device (200) can set a search range for acquiring data related to the intent. The electronic device (200) can acquire data related to the intent based on searching data acquired through multiple applications within the set search range.

[0129] According to one embodiment, the electronic device (200) may acquire third data stored in the electronic device (200) based on the second data, based on data acquired through multiple applications. For example, the third data stored in the electronic device (200) may include data related to an intent.

[0130] In operation 395, the electronic device (200) may obtain second output data. For example, the electronic device (200) may obtain the second output data based on data related to the intent and the first output data. The electronic device (200) may obtain information missing from the first output data through the data related to the intent. The electronic device (200) may obtain sufficient (or appropriate) second output data to provide a response to a language-based request. For example, the electronic device (200) may input data related to the intent and the first output data into a second trained model. The electronic device (200) may obtain the second output data based on the output of the second trained model.

[0131] According to one embodiment, the electronic device (200) may obtain second output data based on the third data and the first output data. For example, the electronic device (200) may input the third data and the first output data into a second trained model. The electronic device (200) may obtain second output data based on the output of the second trained model.

[0132] In operation 396, the electronic device (200) may provide a response to the language-based request. For example, the electronic device (200) may provide a response to the language-based request using the second output data. For example, the response to the language-based request may include performing a function related to the application, executing the application, and / or answering a question.

[0133] Unlike FIG. 3A, according to operations 391 to 396 of FIG. 3B, the electronic device (200) can obtain first output data that does not include some information that the server has not obtained. To obtain the information that the server has not obtained, the electronic device (200) can obtain data related to the intent. Based on the data related to the intent, the electronic device (200) can obtain appropriate second output data in response to the language-based request. Based on the second output data, the electronic device (200) can provide a response to the language-based request.

[0134] FIG. 4 illustrates examples of functional blocks of an electronic device and a server according to one embodiment.

[0135] Referring to FIG. 4, an electronic device (200) can be connected to a server (400).

[0136] According to one embodiment, the electronic device (200) may include a parameter manager (210), a second trained model (220), a conversation service manager (230), and a client (240).

[0137] For example, the parameter manager (210) of the electronic device (200) may include a data manager (211) and a data classifier (212). The data manager (211) may be used to obtain a plurality of candidate parameters from resources available within the electronic device (200). The data classifier (212) may be used to identify at least one candidate parameter among the plurality of candidate parameters using a second trained model (220). For example, the second trained model (220) may be used to output at least one candidate parameter according to inputs of the plurality of candidate parameters. For example, the second trained model (220) may be configured based on an LLM. For example, the second trained model (220) may be referred to as an eLLM. For example, the second trained model (220) may be included within at least a portion of a language package.

[0138] The specific operations of the parameter manager (210) and the second trained model (220) will be described later in FIG. 6.

[0139] For example, the conversation service manager (230) can be used to perform functions related to a conversational application. The conversation service manager (230) can include an automatic speech recognition (ASR) (231), a natural language understanding (NLU) (232), an action executor (233), and a model interface (234). The ARS (231) can be used to convert speech data into text in sentence units. The NLU (232) can be used to understand and interpret the meaning of the text. The action executor (233) can be used to perform functions related to the electronic device (200) (or an application of the electronic device (200), a service of the electronic device (200)). The model interface (234) can be used to communicate with the second trained model (220) (or control the second trained model (220)). For example, a client (240) can be used to send data to a server (400) or receive data from a server (400).

[0140] According to one embodiment, the server (400) may include an operation manager (410), an orchestrator (420), a function finder (430), and a parameter manager (440).

[0141] For example, the operation manager (410) may be used to manage operations for an interactive application running on the electronic device (200). The operation manager (410) may include an NLU (411), a conversation manager (412), a dispatcher (413), an action executor (414), and a user manager (415). For example, the NLU (411) may be used to understand and interpret the meaning of text. The conversation manager (412) may be used to manage conversations provided in the interactive application. The conversation manager (412) may be used to create and manage conversations in response to user input (e.g., language-based requests). The conversation manager (412) may be used to manage and utilize conversation history and / or context information while the user input is maintained. A dispatcher (413) may be used to identify a target (e.g., an application, a device, or a named entity) for a language-based request. An action executor (414) may be used to execute a function related to the server (400) (or an application of the server (400) or a service of the server (400)). A user manager (415) may be used to manage users of an interactive application.

[0142] For example, the orchestrator (420) may be used to manage input data and / or output data for the first trained model (450). The orchestrator (420) may be used to configure input data for the first trained model (450). The orchestrator (420) may identify a candidate list of functions (e.g., descriptions or parameters for functions) that have a high probability of being performed by the input (or language-based request) through the feature finder (430). The orchestrator (420) may perform a post-processing operation, such as a prompt formatting operation, using the identified candidate lists. The orchestrator (420) may perform an operation to obtain a plan from the first trained model (450) using data obtained according to the post-processing operation, such as the prompt formatting operation. For example, the second trained model (220) may provide, as a result (e.g., output data), a plan for functions to be performed on an input (or language-based request) based on data (or information) received from the orchestrator (420).

[0143] According to one embodiment, the plan acquired through the first trained model (210) may include information regarding parameters required for the process of performing functions. For example, required parameter information may be missing depending on information not included in the user input. For example, to perform a function for sending a message (e.g., sendMessage), a parameter for the message content (e.g., messageContent) and a parameter for the recipient (e.g., targetContact) may be required. For example, when a language-based request such as “Send me a text saying the weather is nice” is received, among the parameters required for performing the function for sending a message (e.g., sendMessage), the parameter for the message content “the weather is nice” (e.g., messageContent) may be present, but the parameter for the recipient (targetContact) may be missing.

[0144] As described above, before requesting the missing parameter through a prompt to the user to obtain the missing parameter, the missing parameter can be identified (or selected) using databases associated with the server (400). For example, the data manager (441) included in the parameter manager (440) can identify (or select) the missing parameter based on at least one of the personal database (491), the entity database (492), and / or the device database (493). As an example,

[0145] For example, when a language-based request such as "Change channel" is received, it can be identified that the user has requested control of the TV based on the function for changing the channel (e.g., changeChannel). The target of the control can be identified as a TV. The electronic device (200) can provide a function (or service) for changing the channel of the TV based on the output data acquired from the server. For example, when a language-based request such as "Change channel" is received, the server (400) can identify the function for changing the channel (e.g., changeChannel). The server (400) can identify (or select) a missing parameter based on at least one of the base (491), the entity database (492), and / or the device database (493) using the data manager (441). The server (400) can obtain TV information registered to the user through the IoT service as the missing parameter. For example, the server (400) can identify channel information to be changed and provide the identified channel information as output data to the electronic device (200). The electronic device (200) can provide a function (or service) for changing the channel of the TV based on the output data obtained from the server.

[0146] According to one embodiment, if the missing parameter is not identified (or selected) through the parameter manager (440) as described above, the electronic device (200) may transmit at least some of the data for the language-based request and / or information (e.g., intent) obtained from the server (400).

[0147] According to one embodiment, the electronic device (200) may obtain a plurality of candidate parameters from resources (e.g., personalized data) available within the electronic device (200) based on at least some of the data for the language-based request and / or information (e.g., intent) obtained from the server (400). The electronic device (200) may identify at least one parameter among the plurality of candidate parameters and transmit the parameter to the server (400). The server (400) may obtain output data based on inputting data including input data and at least one candidate parameter into the first trained model (450). The obtained output data may reflect personalized information of the user. The server (400) may transmit the output data to the electronic device (200). The electronic device (200) may use the output data to provide a response to the language-based request.

[0148] In FIG. 4, the first trained model (450) is described as being distinguished from the server (400), but this is exemplary and not limited thereto. The first trained model (450) may also be included in the server (400).

[0149] FIG. 5 illustrates an example of the operation of an electronic device and a server according to one embodiment.

[0150] Referring to FIG. 5, in operation 501, a client (240) of an electronic device (200) may transmit input data including at least a portion of a language-based request to a server (400). For example, the electronic device (200) may receive a language-based request using an interactive application. Based on receiving the language-based request, the client (240) may transmit input data including at least a portion of the language-based request to the server (400).

[0151] In operation 502, the server (400) (e.g., the operation manager (410) of the server (400)) may identify a function for the language-based request. For example, the server (400) (or the operation manager (410)) may use the function finder (430) to identify a candidate list of functions that have a high probability of being performed in response to the language-based request. For example, the server (400) may identify functions that have a high probability of being performed in response to the language-based request.

[0152] For example, the server (400) (or the operation manager (410)) can identify functions that are likely to be performed according to a language-based request in order to reduce the load on the first trained model (450). The server (400) can identify (or select) candidate functions according to the language-based request (or user input) using a feature finder (430) that supports vector-based similarity search. For example, when a language-based request such as “Where was the restaurant for the family gathering last week?” is received, the server (400) can identify candidate functions associated with the language-based request using the feature finder (430). For example, candidate functions such as executing a schedule application (e.g., app_schedule), setting an alarm, and / or searching for a contact can be identified. When N candidate functions are obtained, the feature finder (430) can convey (or transmit) the N candidate functions to the orchestrator (420). The orchestrator (420) can transmit (or send) N candidate functions and receive a plan as output data. For example, the plan can be referenced as information for instructing at least one function to be performed on the electronic device (200) according to a language-based request.

[0153] In operation 503, the server (400) may request a plan according to a language-based request from the first trained model (450). The server (400) may request a plan according to a language-based request based on providing the first trained model (450) with functions that are likely to be performed according to the language-based request, intents according to the language-based request, and / or parameters according to the language-based request.

[0154] In operation 504, the first trained model (450) may transmit information about missing parameters to the server (400) if the information (or parameters) included in the language-based request is not sufficient to obtain the plan (or output data). The server (400) may receive information about missing parameters from the first trained model (450).

[0155] In operation 505, the server (400) (or parameter manager (440)) can search for missing parameters within databases connected to the server (400) (e.g., personal database (491), entity database (492), or device database (493)). For example, the server (400) can identify (or select) the missing parameter based on at least one of the databases connected to the server (400). If the server (400) identifies (or selects) the missing parameter based on at least one of the databases connected to the server (400), the server (400) can perform operation 513.

[0156] For example, if the server (400) cannot identify (or select) a missing parameter based on at least one of the databases connected to the server (400), the server (400) may perform operation 506.

[0157] In one embodiment, the server may transmit input data and at least one parameter stored in the server to the electronic device (200), and request transmission of any missing parameters. For example, the server may transmit input data and at least one parameter stored in the server, and request transmission of any missing parameters, so that additional parameters can be identified through information stored in the electronic device (200) (e.g., personalized information) based on identifying the missing parameters.

[0158] In operation 506, the server (400) may transmit information including an intent included in the language-based request to the electronic device (200) (e.g., the client (240)). For example, the information may include parameter(s) included in the language-based request. For example, the information may include information regarding the type of parameter used to obtain output data. The electronic device (200) may identify the type of parameter requested from the server (400) based on the information including the intent included in the language-based request.

[0159] At operation 507, the client (240) of the electronic device (200) may request the parameter manager (210) to retrieve available resources (550) within the electronic device (200).

[0160] In operations 508 and 509, the parameter manager (210) (or data manager (211)) may search for resources (550) available within the electronic device (200). The parameter manager (210) may search for resources available within the electronic device (200) to obtain parameters related to the intent. The parameter manager (210) may identify (or obtain) a plurality of candidate parameters within the resources available within the electronic device (200).

[0161] In some embodiments, if the parameter manager (210) fails to identify multiple candidate parameters, it may request candidate parameters from the user. Based on the response received from the user, the parameter manager (210) may generate at least one candidate parameter.

[0162] In operation 510, the parameter manager (210) (or data classifier (212)) can identify at least one candidate parameter among a plurality of candidate parameters through filtering. For example, the parameter manager (210) (or data classifier (212)) can identify at least one candidate parameter among a plurality of candidate parameters using a second trained model (220). The plurality of candidate parameters can be input to the second trained model (220). The parameter manager (210) (or data classifier (212)) can identify at least one candidate parameter based on the output of the second trained model (220). The electronic device (200) can identify at least one candidate parameter as at least one parameter to be provided to the server (400).

[0163] According to one embodiment, the electronic device (200) can remove parameters related to limited personal information of the user from at least one candidate parameter. For example, the limited personal information of the user may include contact information, credentials, passwords, account information, and / or card information. For example, the electronic device (200) can perform encryption on at least one candidate parameter. Based on the encryption of at least one candidate parameter, the electronic device (200) can identify at least one parameter.

[0164] In operation 512, the electronic device (200) (e.g., the client (240) of the electronic device (200)) may transmit data including at least one parameter to the server (400). The server (400) may receive data including at least one parameter.

[0165] In operation 513, even if the server (400) receives data including at least one parameter, it can identify whether there are parameters (or information) required to obtain output data according to the input data. If there are parameters (or information) required to obtain output data according to the input data, the server (400) can prompt the user for the required parameters (or information). If there are no parameters (or information) required to obtain output data according to the input data, operation 513 can be omitted.

[0166] In operation 514, the server (400) may request a new plan from the first trained model (450). For example, the server (400) may request a new plan from the first trained model (450) by transmitting input data including at least a portion of the language-based request and data including at least one parameter obtained according to operation 512 to the first trained model (450).

[0167] For example, the server (400) may input data including at least a portion of a language-based request and data including at least one parameter to the first trained model (450).

[0168] In operation 515, the server (400) may obtain output data based on the output of the first trained model (450). For example, the output data may include a plan for a language-based request.

[0169] In operation 516, the server (400) may transmit output data to an electronic device (200) (e.g., a client (240)). The electronic device (200) may use the output data to provide a response to a language-based request.

[0170] Figure 6 illustrates an example of the operation of a server according to one embodiment.

[0171] Referring to FIG. 6, the operation of the server (400) described in FIG. 6 may be related to operations 501 to 506 of FIG. 5.

[0172] According to one embodiment, the operation manager (410) may receive input data including at least a portion of a language-based request from the electronic device (200) (or a client (240) of the electronic device (200). The operation manager (410) may use the function finder (430) to identify candidate functions according to the language-based request. For example, the operation manager (410) may use the function finder (430) to identify a candidate list of functions (or candidate functions) that have a high possibility of being performed according to the language-based request. For example, the server (400) may identify functions (or candidate functions) that have a possibility of being performed according to the language-based request. The operation manager (410) may forward (or transmit) the candidate functions to the orchestrator (420).

[0173] In one embodiment, the orchestrator (420) may request the first trained model (450) to generate a plan. Based on the plan being generated (or returned) from the first trained model (450), the operation manager (410) may obtain the generated plan. The operation manager (410) may identify whether parameters related to functions to be performed according to the plan are missing. If parameters required for functions to be performed according to the plan are missing, the parameter manager (440) may collect (or search for) the required parameters from a database connected to the server (400) (e.g., the personal database (491), the entity database (492), or the device database (493) of FIG. 4 ). For example, if the operation manager (410) does not have the parameters required for functions to be performed according to the plan, it can transfer the already identified parameters to the parameter manager (440) and request the required parameters from the parameter manager (440).

[0174] For example, if the required parameter is device information, the server (400) (or parameter manager (440))<Device_Type:string> A format such as this can be transmitted to the database. For example, if the required parameter is a device name, the server (400) (or parameter manager (440))<Device_Name:string> You can pass the same format to the database.

[0175] For example, the parameter manager (440) can search a database synchronized with the server (400) (e.g., a personal database (491), an entity database (492), or a device database (493) of FIG. 4) using an information query for a required type. The parameter manager (440) can obtain the required parameter if the database contains information that matches the information query.

[0176] For example, when a language-based request such as “turn on the clicker” is identified, the orchestrator (420) can obtain {intent: turn_on_device, function: ()[target_device], (turn on)[turn_on_device]} as a plan according to the first trained model (450) and obtain (clicker)[device name] as a parameter (or parameter information).

[0177] For example, the orchestrator (420) is a parameter manager (440) that manages the parameters required to perform functions according to the plan.<Device_ID:string> and (clicker)<Device_Name:string> can be transmitted. The parameter manager (440)<Device_Name> By using the databases linked to the server (400),<Device_Name> (or "clicker") can be used to identify information that matches the<Device_Name> linked (or mapped) to<Device_Type> If this 'TV' is identified, then according to the information query<Device_ID> can be returned (or identified, obtained).

[0178] As in the example described above, since the parameter manager (440) only uses the database linked with the server (400), it may not be able to utilize information (or resources) stored in the electronic device (200). Therefore, even after searching the database information using the parameter manager (440), if there are parameters required to perform a function according to a language-based request, the parameters obtained from the server (400) and information about the required parameters may be transmitted to the electronic device (200). The server (400) may obtain the required parameters from the electronic device (200) and provide output data (or a plan) to the electronic device (200). For example, the electronic device (200) may obtain at least one parameter and transmit the at least one parameter to the server (400). The operation of the electronic device for transmitting at least one parameter to the server (400) will be described below with reference to FIGS. 7 and 8 .

[0179] FIG. 7 illustrates an example of operation of an electronic device according to one embodiment.

[0180] Referring to FIG. 7, the server (400) can transmit to the electronic device (200) information including an intent included in a language-based request (or information including a language-based request), information regarding identified parameter(s), and information regarding required parameter(s).

[0181] According to one embodiment, the data manager (211) of the parameter manager (210) can list information about at least one application, at least one service, and / or at least one sensor stored (or installed) in the electronic device (200). For example, the data manager (211) can identify resources (550) available within the electronic device (200). For example, the resources (550) available within the electronic device (200) can include resources (551) acquired based on at least one piece of hardware (e.g., a sensor, an illumination sensor, a communication circuit) and resources (552) acquired based on at least one piece of software (e.g., an application, a service). For example, the resources (551) acquired based on at least one piece of hardware can include illumination data and / or location data. For example, resources (552) acquired based on at least one software application may include data stored through a calendar application, data stored through a contact application, data stored through a file application, and / or data stored through a message application. For example, resources available within an electronic device (200) may be configured differently depending on the state (or situation) of the electronic device (200).

[0182] For example, the format converter (252) may be used to identify (or query) information about available applications, available services, and / or available sensors. For example, since each application, service, and / or sensor processes data differently, the same data may be stored in different formats. Therefore, the format converter (252) may be used to convert a query so as to identify information (or parameters) about the application, service, and / or sensor. The mapping repository (251) may be used to store mapping information about the format of data received from the server (400) and the formats of data stored through the applications, services, and / or sensors of the electronic device (200). The format converter (252) may convert the format of the required parameter(s) received from the server (400) based on the mapping information stored in the mapping repository (251). The format converter (252) may generate a query (or query statement) using parameter(s) configured based on another format based on the mapping information. The format converter (252) can transmit (or send) the generated query (or query statement) to the data collector (254).

[0183] For example, the coverage scaler (253) may be used to set a search scope for resources available within the electronic device (200). For example, the search scope may be set based on a policy (e.g., a resource provider's policy) and / or logic (e.g., business logic). The coverage scaler (253) may transmit (or transmit) the set search scope to the data collector (254).

[0184] For example, the data collector (254) can retrieve resources (550) available to the electronic device (200) using a query obtained based on information received from the format converter (252) and the coverage scaler (253). For example, the data collector (254) can use the query to search for and / or collect information about applications, services, and / or sensors.

[0185] For example, based on a language-based request such as "When is the family gathering next week?", the electronic device (200) can obtain parameter information in the following format from the server (400).

[0186]

[0187] When parameter information such as Table 1 described above is received, the electronic device (200) can change the format of the received parameter information to the format of information stored through the calendar application as shown in Table 2 below, based on the mapping information according to the mapping storage (251).

[0188]

[0189] Referring to Table 2, the electronic device (200) can change the format of parameter information received from the server (400). For example, the electronic device (200) can change the format "period" to the format "duration." The electronic device (200) can change the format "startDate" to the format "dtstart."

[0190] For example, the data collector (254) can generate a query based on parameter information configured based on the changed format as described above. The table below shows examples of generated queries.

[0191]

[0192] Referring to Table 3, a query can be generated based on an application, a service, and / or an application programming interface (API) provided by a service. For example, a language-based request such as “When is the family gathering next week?” can include an intent for asking about the time and an intent for asking about the date. The server (400) can request “When is the family gathering next week?” and “Duration next week” to the electronic device (200). The electronic device (200) can request date information and location information together. The electronic device (200) can request date information and / or location information from various applications (e.g., a messaging application, a calendar application). The electronic device (200) can determine (or filter) information to be transmitted to the server (400) and transmit location information or attendee information that the user did not request together to the server (400).

[0193] According to one embodiment, the information (or parameters) acquired through the data collector (254) may be acquired through a specified format (e.g., JSON format). Since the information acquired through the data collector (254) includes raw data, the electronic device (200) may perform a filtering operation to identify at least one parameter to be transmitted to the server (400).

[0194] For example, when text information and call information for a specific person are searched, dozens of records may be searched, so the electronic device (200) can filter the searched information.

[0195] According to one embodiment, the electronic device (200) may perform a filtering process to identify at least one candidate parameter among a plurality of candidate parameters. The filtering process performed in the electronic device (200) may be referred to as a data sorting filter (DSF) process. For example, the electronic device (200) may use a second trained model (220) to perform the filtering process. According to an embodiment, the second trained model (220) may be configured based on at least one of a rule-based model, a pattern-based model, a deep-based model, and / or an LLM.

[0196] For example, the second trained model (220) may be configured based on a rule-based model and / or a pattern-based model. The electronic device (200) may identify information (or parameters) that satisfy conditions based on rules or pattern matching. If the second trained model (220) is configured based on a rule-based model and / or a pattern-based model, parameter information must be converted to a format appropriate for the desired format, and tuning may be required based on various conditions.

[0197] For example, the second trained model (220) may be configured based on a rule-based model and / or a pattern-based model. The electronic device (200) may identify information (or parameters) that satisfy conditions based on rules or pattern matching. If the second trained model (220) is configured based on a rule-based model and / or a pattern-based model, parameter information must be converted to a format appropriate for the desired format, and tuning may be required based on various conditions.

[0198] For example, the second trained model (220) may be configured based on a deep-based model. The electronic device (200) may use the second trained model (220) configured based on the deep-based model to select appropriate parameters (or values). If the second trained model (220) is configured based on the deep-based model, data learning work is required, and performance may not be high.

[0199] For example, the second trained model (220) may be constructed based on LLM. The electronic device (200) may use the second trained model (220) constructed based on a deep-based model to select appropriate parameters (or values). If the second trained model (220) is constructed based on a deep-based model, data learning work is required, and performance may not be high.

[0200] For example, in order to select the most appropriate value considering the context from scattered information, the second trained model (220) can be configured based on LLM. If the second trained model (220) is configured based on LLM, the prompt builder (261) can request the second trained model (220) to select the most appropriate value from the acquired information by utilizing a prompt set for each type of required parameter. For example, if the required parameter is ' <location>', the electronic device (200) receives from the prompt builder (261) among the parameter types, ' <location>' can identify a prompt for. The electronic device (200) can use the identified prompt to obtain at least one candidate parameter among a plurality of candidate parameters.

[0201] For example, if a prompt configured according to the form and type of information to be acquired is prepared (or stored), the electronic device (200) can select an appropriate prompt and utilize the second trained model (220) according to the prompt. In some embodiments, if an appropriate prompt is not prepared (or stored), the electronic device (200) can also request the second trained model (220) to generate a prompt based on the stored format.

[0202] As described above, the electronic device (200) can collect necessary information (or parameters) and identify at least one candidate parameter among a plurality of candidate parameters using the second trained model (220). For example, by filtering information (e.g., a plurality of candidate parameters) in the electronic device (200) and transmitting the filtered information (e.g., at least one candidate parameter) to the server (400), an increase in data communication costs and leakage of personal information can be reduced.

[0203] According to one embodiment, the electronic device (200) can receive information about identified parameter(s) and information about required parameter(s) from the server (400). Since the format in which information is stored may be different for each application, service, and sensor, the parameter manager (210) can change the parameter information (e.g., information about identified parameter(s) and information about required parameter(s)) into a format that can be processed by the application, service, and sensor.

[0204] For example, the format converter (252) can change parameter information into a format that can be processed by the application, service, and sensor by using the mapping information stored in the mapping storage (251) according to the format required by each application, service, and sensor.

[0205] For example, the coverage scaler (253) can set a search range for resources available within the electronic device (200). For example, the information to be searched may be <time>If so, a time-based search range can be set, such as a time period before the last n hours or a time period within n hours before or after the time when the language-based request was received. The information you want to search for <date>If the information you are searching for is within a date range of n days / weeks / months before or after the time the language-based request was received, a date-based search range can be set. <location>In this case, the search range can be set, such as a geographical range within a radius of X km from the location of the current electronic device (200) or a geographical range within a radius of X km from the requested target area.

[0206] For example, the data collector (254) can search (or inquire) for information (or parameters) within the search range set by the coverage scheduler (253) within the resources (550) available within the electronic device (200), according to the query. Based on the resources (550) available within the electronic device (200), information (or parameters) that satisfy the conditions according to the query can be acquired.

[0207] Through the operation of the electronic device (200) described above, in an environment where the electronic device (200) and the server (400) are interconnected (e.g., a hybrid environment), information about all applications, all sensors, and all services of the electronic device (200), which is a user's personal device, including messages and notes, can be retrieved. Accordingly, the electronic device (200) (or the server (400)) can provide an accurate response to a language-based request to the user. According to one embodiment, personal information can be transmitted to the server (400) with a minimum of word information or phrase information. According to the above-described embodiment, there is an effect that the personal information of the user of the electronic device (200) can be protected.

[0208] According to one embodiment, the format of information (or parameter information) obtained from resources (550) available within the electronic device (200) (e.g., data stored through a calendar application) may be set as shown in Tables 4 and 5 below.

[0209]

[0210]

[0211] According to one embodiment, information obtained from resources (550) available within the electronic device (200) may be extensive and include a large number of values. Transmitting the obtained information to the server (400) increases overhead and may lead to personal information leakage. Therefore, to reduce transmission load and protect personal information, the obtained information may be filtered. The filtering process described below may be referred to as the intelligence filtering (IF) process (or DSF process).

[0212] In one embodiment, different techniques may be applied to the IF process, depending on the purpose. For example, if the IF process is performed based on a rule model, data can be classified based on information type and pattern matching. For example, if the IF process is performed based on a deep model, data can be classified based on a specified number of candidate values ​​with high accuracy selected based on reference information.

[0213] For the rule models and / or deep models described above, it can be difficult to see the relationships and context between the user's input and each parameter. Consequently, performance limitations may arise. Therefore, an LLM (e.g., an on-device LLM) may be used for the IF process. An LLM embedded within the electronic device (200) may be used to select the most appropriate value for this wide range. For example, the LLM described below may be configured within the electronic device (200).

[0214] In one embodiment, due to limitations in the processing capabilities of the electronic device (200), a high-complexity LLM may not be used. Therefore, a prompt may be generated (or identified) by the prompt builder (261) so that the acquired information can be processed through the LLM. For example, the prompt builder (261) may utilize pre-stored prompts depending on the type of parameter. For example, <location>To identify parameters for , prompts such as those in the table below can be used.

[0215]

[0216] FIG. 8 illustrates an example of operation of an electronic device according to one embodiment.

[0217] Referring to FIG. 8, the electronic device (200) may identify at least one candidate parameter among a plurality of candidate parameters using the second trained model (220) based on a prompt. If the size of the input data of the second trained model (220) is larger than the sizes of the plurality of candidate parameters, at least one candidate parameter may not be acquired with a single input. Accordingly, the electronic device (200) may divide the plurality of candidate parameters into one or more parameter sets based on a reference size (e.g., batch size) that can be processed by the second trained model (220). The electronic device (200) may sequentially input each of the one or more parameter sets into the second trained model (220). The electronic device (200) may identify at least one candidate parameter based on the outputs of the second trained model (220).

[0218] According to one embodiment, the electronic device (200) may use the post-processor (262) to remove parameters related to restricted personal information from at least one candidate parameter identified through the second trained model (220). For example, the restricted personal information may include contact information, credentials, passwords, account information, and / or card information. The electronic device (200) may use the post-processor (262) to remove parameters related to restricted personal information from at least one candidate parameter so that they are not transmitted to the server (400).

[0219] According to one embodiment, at least one candidate parameter may be converted into a format that can be processed by the server (400) and then transmitted to the server (400). For example, the electronic device (200) may perform encryption on the at least one candidate parameter. The electronic device (200) may transmit the encrypted at least one candidate parameter to the server (400). The server (400) may generate (or obtain) output data according to the at least one candidate parameter and discard the at least one candidate parameter without storing it.

[0220] According to one embodiment, the electronic device (200) may provide candidate responses in response to a language-based request based on output data. For example, the candidate responses may be provided to the user, and at least one of the candidate responses may be selected by the user.

[0221] For example, if there is one candidate response, the electronic device (200) may perform a function for the candidate response or ask the user whether to perform a function for the candidate response.

[0222] For example, if there is no candidate response, the electronic device (200) may request additional information from the user and obtain output data based on the response obtained based on the request.

[0223] According to one embodiment, the electronic device (200) can identify a language-based request based on multiple intents (or multiple turns). The electronic device (200) can also search for resources available within the electronic device (200) to obtain the parameters required for each intent according to the operations described above for the language-based request based on multiple intents (or multiple turns).

[0224] For example, the electronic device (200) can identify a language-based request such as "See where we're having dinner tonight and text the attendees." The language-based request can include two intents. The electronic device (200) can obtain "Tonight's schedule in Seoul - Restaurant A" from data stored through a calendar application. The electronic device (200) can transmit parameter information including "Kim Sam-seong" and "Hong Gil-dong" along with "Tonight's schedule in Seoul - Restaurant A" to the server (400).

[0225] In one embodiment, the server (400) may generate (or obtain) 'Seoul-A Restaurant' in response to 'Where are you planning to eat dinner tonight?' The server (400) may respond to the language-based request with 'Seoul-A Restaurant', and generate output data instructing to compose a message saying 'Dinner Date Seoul-A Restaurant' and send the message to 'Kim Sam-seong' and 'Hong Gil-dong'. The electronic device (200) may display a response message indicating that the dinner location tonight is Seoul-A Restaurant based on the output data obtained from the server (400), and display an object indicating that the message 'Dinner Date Seoul-A Restaurant' has been sent to 'Kim Sam-seong' and 'Hong Gil-dong'.

[0226] FIGS. 9A and 9B illustrate examples of operation of an electronic device according to one embodiment.

[0227] Referring to FIG. 9A, the electronic device (200) can display a user interface (900) of an interactive application through a display (202). The electronic device (200) can receive a language-based request. The electronic device (200) can display an object (901) representing the language-based request within the user interface (900). For example, the electronic device (200) can receive a language-based request, such as "Tell me my schedule and weather for tomorrow."

[0228] According to one embodiment, the electronic device (200) may transmit input data including at least a portion of a language-based request, such as “Tell me my schedule and weather for tomorrow,” to the server (400). The electronic device (200) may receive information including an intent included in the language-based request. The electronic device (200) may transmit parameters obtained based on data (910) stored through a calendar application to the server (400). For example, the electronic device (200) may transmit parameters regarding tomorrow’s schedule to the server (400). The electronic device (200) may use the second trained model (220) to identify that the current user location is Suwon and the user location at 2 PM tomorrow is Seoul. The electronic device (200) may transmit information (or parameter information) indicating that the current user location is Suwon and the user location at 2 PM tomorrow is Seoul to the server (400). The electronic device (200) can transmit information (or parameter information) to the server (400) indicating that there is a meeting with Jane at Seoul City Hall tomorrow at 2:00 PM. The server (400) can obtain tomorrow's weather information from a weather server. Based on the information (or parameter information) received from the electronic device (200) and the tomorrow's weather information obtained from the weather server, the server (400) can obtain output data using the first trained model (450).

[0229] According to one embodiment, the electronic device (200) may, based on receiving output data from the server (400), display an object (902) representing a response such as, "I have a meeting with Jane at Seoul City Hall at 2 PM tomorrow. The weather in Seoul at 2 PM is 17 degrees, cloudy, windy, and chilly." using the output data.

[0230] Referring to FIG. 9B, the electronic device (200) can display a user interface (950) of an interactive application through the display (202). The electronic device (200) can receive a language-based request. The electronic device (200) can display an object (951) representing the language-based request within the user interface (950). For example, the electronic device (200) can receive a language-based request, such as “Tell me my schedule for tomorrow.” The electronic device (200) can transmit input data including at least a portion of the language-based request, such as “Tell me my schedule for tomorrow,” to the server (400). The electronic device (200) can receive information including an intent included in the language-based request. The electronic device (200) can transmit acquired parameters to the server (400) based on data (960) stored through the calendar application and data (970) stored through the messenger application. For example, the electronic device (200) can transmit parameters for today's schedule and parameters for tomorrow's schedule to the server (400). The electronic device (200) can transmit information (or parameter information) indicating that there is a dinner with Jun at 7 PM today and information (or parameter information) indicating that there is a meeting with Jane at Seoul City Hall at 2 PM tomorrow to the server (400). The server (400) can obtain output data using the first trained model (450) based on the information (or parameter information) received from the electronic device (200). Based on receiving the output data from the server (400), the electronic device (200) can display an object (952) indicating a response such as, "I have a meeting with Jane at Seoul City Hall tomorrow at 2 PM saved in my calendar and I have made a dinner appointment with Jun at 7 PM in the chat app."

[0231] In the above-described embodiments, parameter information has been described as representing sentences. However, this is for convenience of explanation and is not limited thereto. Parameter information may be transmitted as composed of individual parameters (e.g., parameters representing locations, parameters representing objects, and parameters representing time).

[0232] FIGS. 10A and 10B illustrate examples of operation of an electronic device according to one embodiment.

[0233] Referring to FIG. 10A, the electronic device (200) can display a user interface (1000) of an interactive application through a display (202). The electronic device (200) can receive a language-based request. The electronic device (200) can display an object (1001) representing the language-based request within the user interface (1000). For example, the electronic device (200) can receive a language-based request, such as "Tell me the weather in July."

[0234] According to one embodiment, the electronic device (200) may transmit input data including at least a portion of a language-based request, such as “Tell me the weather in July,” to the server (400). The electronic device (200) may receive information including an intent included in the language-based request. The electronic device (200) may transmit parameters obtained based on data (1010) stored through a calendar application to the server (400). For example, the electronic device (200) may transmit parameters for a July schedule to the server (400). The electronic device (200) may use the second trained model (220) to identify (or predict) that the current user location is Seoul and the user’s location in July is Bali. For example, the electronic device (200) may transmit information (or parameter information) indicating that the current user location is Seoul and the user’s location in July is Bali to the server (400). The electronic device (200) can transmit information (or parameter information) indicating that the current location of the user is Seoul and information (or parameter information) indicating that the location of the user in July is Bali to the server (400). The server (400) can obtain current weather information for Seoul and weather information for Bali in July from a weather server. The server (400) can obtain output data using the first trained model (450) based on the information (or parameter information) received from the electronic device (200) and the weather information obtained from the weather server (e.g., current weather information for Seoul and weather information for Bali in July).

[0235] According to one embodiment, the electronic device (200) may, based on receiving output data from a server (400), display an object (1003) representing a response to a language-based request using the output data.

[0236] Referring to FIG. 9B, the electronic device (200) can display a user interface (1050) of an interactive application through a display (202). The electronic device (200) can receive a language-based request. The electronic device (200) can display an object (1051) representing the language-based request within the user interface (1050). For example, the electronic device (200) can receive a language-based request, such as “Tell me the weather tomorrow.” The electronic device (200) can transmit input data including at least a portion of the language-based request, such as “Tell me the weather tomorrow,” to a server (400). The electronic device (200) can receive information including an intent included in the language-based request. The electronic device (200) can transmit acquired parameters to the server (400) based on location data (1060) according to a location-based service. For example, the electronic device (200) may transmit parameters regarding the user's location over time to the server (400). The electronic device (200) may transmit information (or parameter information) indicating that the user is located in Suwon from 9:00 AM to 6:00 PM and that the user is located in Seoul from 7:00 PM to 8:00 PM to the server (400). The information (or parameter information) may be acquired through the second trained model (220).

[0237] According to one embodiment, the server (400) may obtain output data using the first trained model (450) based on information (or parameter information) received from the electronic device (200). Based on receiving the output data from the server (400), the electronic device (200) may display an object (1053) representing a response to a language-based request using the output data.

[0238] In the above-described embodiments, parameter information has been described as representing sentences. However, this is for convenience of explanation and is not limited thereto. Parameter information may be transmitted as composed of individual parameters (e.g., parameters representing locations, parameters representing objects, and parameters representing time).

[0239] FIGS. 11A, 11B, and 11C illustrate examples of operation of an electronic device according to one embodiment.

[0240] Referring to FIG. 11A, the electronic device (200) can display a visual object (1101) through an image application. For example, the electronic device (200) can display the visual object (1101) through the display (202) within a user interface (1121) for the image application. While the visual object (1101) is displayed, an interactive application can be executed. The electronic device (200) can receive a language-based request, such as “call this number,” through the interactive application. Based on the language-based request, the electronic device (200) can transmit input data including at least a portion of the language-based request to the server (400).

[0241] According to one embodiment, the server (400) may identify that parameters related to a phone number are required. The server (400) may transmit information for requesting parameters related to the phone number to the electronic device (200). The electronic device (200) may obtain a plurality of candidate parameters related to the phone number using the data manager (211). The electronic device (200) may identify at least one candidate parameter among the plurality of candidate parameters using the second trained model (220). For example, the electronic device (200) may prioritize identifying a parameter obtained through an application running in the foreground. The electronic device (200) may identify the phone number displayed on the visual object (1101) as the most appropriate number. The electronic device (200) may make a call using the identified phone number. According to an embodiment, when two or more phone numbers are identified, the electronic device (200) may provide candidate phone numbers and perform a prompt process for receiving a user's selection.

[0242] For example, the electronic device (200) can obtain text information (1102) within the currently displayed photo. The electronic device (200) can obtain a plurality of candidate parameters included in the text information (1102). For example, the electronic device (200) can obtain a parameter indicating an address, a parameter indicating a phone number, a parameter indicating an email address, and a parameter indicating a URL. The electronic device (200) can obtain a parameter indicating a phone number among the plurality of candidate parameters using the second trained model (220). The electronic device (200) can transmit the parameter indicating the phone number to the server (400). The electronic device (200) can receive output data from the server (400) that instructs to make a call to the phone number identified in the image. The electronic device (200) can make a call to the phone number identified in the image based on the output data. For example, the electronic device (200) can execute a phone application based on the output data. The electronic device (200) can display a user interface (1122) for a phone application. The electronic device (200) can use the phone application to make a call to a phone number identified in the image.

[0243] Referring to FIG. 11B, the electronic device (200) can display a visual object (1101) through an image application. For example, the electronic device (200) can display the visual object (1101) through the display (202) within a user interface (1121) for the image application. While the visual object (1101) is displayed, an interactive application can be executed. For example, the electronic device (200) can display a user interface (1110) for the interactive application through the display (202). The electronic device (200) can receive a language-based request, such as "Put this on my calendar," through the interactive application. Based on receiving the language-based request, the electronic device (200) can display an object (1111) representing the language-based request. The electronic device (200) can transmit input data including at least a portion of the language-based request to the server (400) based on the language-based request.

[0244] For example, the server (400) may identify that parameters regarding date, time, location, and / or attendees are required. The server (400) may transmit information to the electronic device (200) to request parameters regarding date, time, location, and / or attendees. The electronic device (200) may use the data manager (211) to obtain a plurality of candidate parameters regarding date, time, location, and / or attendees. The electronic device (200) may use the second trained model (220) to identify at least one candidate parameter among the plurality of candidate parameters.

[0245] For example, the electronic device (400) can identify the user's schedule information (1105) through a calendar application. The electronic device (400) can identify parameters regarding the date, time, and attendees.

[0246] For example, the electronic device (200) can identify parameters acquired through an application running in the foreground. The electronic device (200) can identify a location displayed in a visual object (1101) as the most suitable location. According to an embodiment, if two or more locations are identified, the electronic device (200) can provide candidate locations and perform a prompt process to receive a user's selection.

[0247] For example, the electronic device (200) can obtain text information (1102) within a currently displayed photo. The electronic device (200) can obtain a plurality of candidate parameters included in the text information (1102). For example, the electronic device (200) can obtain a parameter indicating an address, a parameter indicating a phone number, a parameter indicating an email address, and a parameter indicating a URL. The electronic device (200) can obtain a parameter indicating a place (e.g., a store name) among the plurality of candidate parameters using the second trained model (220). The electronic device (200) can transmit the parameter indicating the place (e.g., the store name) to the server (400).

[0248] According to one embodiment, the electronic device (400) may transmit parameters regarding the date, time, and attendees identified based on the user's schedule information (1105) to the server (400). The electronic device (400) may transmit parameters indicating a location (e.g., a store name) to the server (400).

[0249] According to one embodiment, the electronic device (200) may receive output data from the server (400) instructing to add a location identified in the image to the user's schedule. Based on the output data, the electronic device (200) may add the location identified in the image (e.g., the visual object (1101)) to the user's schedule. For example, based on the output data, the electronic device (200) may display an object (1112) indicating that the schedule has been saved within a user interface (1110) for an interactive application. Based on the output data, the electronic device (200) may display a user interface (1113) for a calendar application indicating the user's changed schedule within the user interface (1110).

[0250] Referring to FIG. 11C, the operation of the electronic device (200) related to the operation of FIG. 11A is described in FIG. 11C. The electronic device (200) may request analysis of an image from the gallery application (1171), the camera application (1172), and / or the screenshot function (1173) using a capsule (1151) for performing an artificial intelligence function. Data according to the analysis result of the image may be obtained through an intelligent framework (1180). The intelligent framework (1180) may provide the data to the capsule (1151) for performing artificial intelligence through the gallery application (1171). The electronic device (200) may use the capsule (1151) for performing an artificial intelligence function, and may use the capsules (1152, 1153) for performing a phone function to make a call to a phone number identified in the image through a phone application (1174). Although the operations of the electronic device (200) related to the embodiment of FIG. 11a are described in FIG. 11c, they are not limited thereto. The electronic device (200) can perform operations related to the embodiment of FIG. 11b.

[0251] Figure 12 illustrates an example of the operation of an electronic device according to one embodiment.

[0252] Referring to FIG. 12, the electronic device (200) can display a user interface (1200) of an interactive application through a display (202). The electronic device (200) can receive a language-based request. The electronic device (200) can display an object (1201) representing the language-based request within the user interface (1200). For example, the electronic device (200) can receive a language-based request, such as, "Send me a text saying I'm going to be late and find me a place that sells cake."

[0253] The electronic device (200) can transmit input data including at least a portion of a language-based request, such as “Send a text saying I’m going to be late and find a place that sells cake,” to the server (400). The electronic device (200) can receive information including an intent included in the language-based request. The electronic device (200) can obtain at least one parameter (1210) using the second trained model (220) based on searching for resources available within the electronic device (200). The at least one parameter (1210) can include a parameter indicating a name (1211), a parameter indicating a current location (1212), and a parameter indicating an appointment place (1213). The electronic device (200) can transmit the at least one parameter (1210) to the server (400).

[0254] The server (400) can obtain output data using the first trained model (450) based on information and input data including at least one parameter (1210) received from the electronic device (200). Based on receiving the output data from the server (400), the electronic device (200) can display an object (1203) representing a response to a language-based request using the output data.

[0255] Figure 13 is a block diagram illustrating an integrated intelligence system according to one embodiment.

[0256] Referring to FIG. 13, an integrated intelligence system (10) of one embodiment may include a user terminal (1300), an intelligent server (1400), and a service server (1500).

[0257] A user terminal (1300) of one embodiment (e.g., electronic device (101) of FIG. 1) may be a terminal device (or electronic device) that can connect to the Internet, and may be, for example, a mobile phone, a smart phone, a personal digital assistant (PDA), a laptop computer, a TV, white goods, a wearable device, an HMD, or a smart speaker.

[0258] According to one embodiment, the user terminal (1300) may include a communication interface (1310), a microphone (1320), a speaker (1330), a display (1340), a memory (1350), and a processor (1360). The components listed above may be operatively or electrically connected to each other.

[0259] According to one embodiment, the communication interface (1310) may be configured to be connected to an external device to transmit and receive data. According to one embodiment, the microphone (1320) may receive sound (e.g., user speech) and convert it into an electrical signal. According to one embodiment, the speaker (1330) may output the electrical signal as sound (e.g., voice). According to one embodiment, the display (1340) may be configured to display an image or video. According to one embodiment, the display (1340) may display a graphical user interface (GUI) of an app (or application program) being executed.

[0260] The display (1340) of one embodiment may be configured to display an image or video. The display (1340) of one embodiment may also display a graphical user interface (GUI) of a running app (or application program). The display (1340) of one embodiment may receive touch input via a touch sensor. For example, the display (1340) may receive text input via a touch sensor in an on-screen keyboard area displayed within the display (1340).

[0261] According to one embodiment, the memory (1350) may store a client module (1351), a software development kit (SDK) (1353), and a plurality of apps (1355). The client module (1351) and the SDK (1353) may constitute a framework (or solution program) for performing general functions. In addition, the client module (1351) or the SDK (1353) may constitute a framework for processing user input (e.g., voice input, text input, touch input).

[0262] According to one embodiment, the memory (1350) may be a program for performing a specified function of the plurality of apps (1355). According to one embodiment, the plurality of apps (1355) may include a first app (1355_1) and a second app (1355_3). According to one embodiment, each of the plurality of apps (1355) may include a plurality of operations for performing a specified function. For example, the plurality of apps (1355) may include at least one of an alarm app, a message app, and a schedule app. According to one embodiment, the plurality of apps (1355) may be executed by the processor (1360) to sequentially execute at least some of the plurality of operations.

[0263] According to one embodiment, the processor (1360) can control the overall operation of the user terminal (1300). For example, the processor (1360) can be electrically connected to a communication interface (1310), a microphone (1320), a speaker (1330), a display (1340), and a memory (1350) to perform a designated operation.

[0264] According to one embodiment, the processor (1360) may also execute a program stored in the memory (1350) to perform a designated function. For example, the processor (1360) may execute at least one of the client module (1351) or the SDK (1353) to perform the following operations for processing user input. The processor (1360) may control the operations of multiple apps (1355), for example, through the SDK (1353). The following operations described as operations of the client module (1351) or the SDK (1353) may be operations executed by the processor (1360).

[0265] According to one embodiment, the client module (1351) can receive user input. For example, the client module (1351) can generate a voice signal corresponding to a user utterance detected through the microphone (1320). Alternatively, the client module (1351) can receive a touch input detected through the display (1340). Alternatively, the client module (1351) can receive a text input detected through a keyboard or a visual keyboard. In addition, the client module (1351) can receive various forms of user input detected through an input module included in the user terminal (1300) or an input module connected to the user terminal (1300). The client module (1351) can transmit the received user input to the intelligent server (1400). According to one embodiment, the client module (1351) can transmit status information of the user terminal (1300) to the intelligent server (1400) together with the received user input. The above status information may be, for example, the execution status information of the app.

[0266] According to one embodiment, the client module (1351) may receive a result corresponding to the received user input. For example, the client module (1351) may receive a result corresponding to the user input from the intelligent server (1400). The client module (1351) may display the received result on the display (1340). Additionally, the client module (1351) may output the received result as audio through the speaker (1330).

[0267] According to one embodiment, the client module (1351) can receive a plan corresponding to the received user input. The client module (1351) can display the results of executing multiple operations of the app according to the plan on the display (1340). For example, the client module (1351) can sequentially display the results of executing multiple operations on the display and output audio through the speaker (1330). As another example, the user terminal (1300) can display only some results of executing multiple operations (e.g., the result of the last operation) on the display and output audio through the speaker (1330).

[0268] According to one embodiment, the client module (1351) may receive a request from the intelligent server (1400) to obtain information necessary to produce a result corresponding to a user input. The information necessary to produce the result may be, for example, status information of the user terminal (1300). According to one embodiment, the client module (1351) may transmit the necessary information to the intelligent server (1400) in response to the request.

[0269] According to one embodiment, the client module (1351) can transmit result information of executing multiple operations according to a plan to the intelligent server (1400). The intelligent server (1400) can confirm that the received user input has been correctly processed through the result information.

[0270] In one embodiment, the client module (1351) may include a voice recognition module. In one embodiment, the client module (1351) may recognize voice inputs that perform limited functions through the voice recognition module. For example, the client module (1351) may execute an intelligent app to process voice inputs to perform organic actions through designated inputs (e.g., "Wake up!").

[0271] According to one embodiment, the intelligent server (1400) can receive information related to user voice input from the user terminal (1300) via a communication network. According to one embodiment, the intelligent server (1400) can convert data related to the received voice input into text data. According to one embodiment, the intelligent server (1400) can generate a plan for performing a task corresponding to the user voice input based on the text data.

[0272] In one embodiment, the plan may be generated by an artificial intelligence (AI) system. The AI ​​system may be a rule-based system, a neural network-based system (e.g., a feedforward neural network (FNN) or a recurrent neural network (RNN)), or a combination of the above or a different AI system. In one embodiment, the plan may be selected from a set of predefined plans or may be generated in real time in response to a user request. For example, the AI ​​system may select at least one plan from a plurality of predefined plans.

[0273] According to one embodiment, the intelligent server (1400) may transmit the results calculated according to the generated plan to the user terminal (1300), or may transmit the generated plan to the user terminal (1300). According to one embodiment, the user terminal (1300) may display the results calculated according to the plan on a display. According to one embodiment, the user terminal (1300) may display the results of executing an operation according to the plan on a display.

[0274] An intelligent server (1400) of one embodiment may include a front end (1410), a natural language platform (1420), a capsule database (1430), an execution engine (1440), an end user interface (1450), a management platform (1460), a big data platform (1470), and an analytic platform (1480).

[0275] According to one embodiment, the front end (1410) can receive user input from a user terminal (1300). The front end (1410) can transmit a response corresponding to the user input.

[0276] According to one embodiment, the natural language platform (1420) may include an automatic speech recognition module (ASR module) (1421), a natural language understanding module (NLU module) (1423), a planner module (1425), a natural language generator module (NLG module) (1427), and a text to speech module (TTS module) (1429).

[0277] According to one embodiment, the automatic speech recognition module (1421) can convert voice input received from the user terminal (1300) into text data. According to one embodiment, the natural language understanding module (1423) can use the text data of the voice input to determine the user's intention. For example, the natural language understanding module (1423) can perform syntactic analysis or semantic analysis on user input in the form of text data to determine the user's intention. According to one embodiment, the natural language understanding module (1423) can use linguistic features (e.g., grammatical elements) of morphemes or phrases to determine the meaning of words extracted from the user input, and can match the meaning of the determined words to the intent to determine the user's intent. The natural language understanding module (1423) can obtain intent information corresponding to the user's utterance. The intent information can be information indicating the user's intent determined by interpreting text data. Intent information may include information indicating an action or function that a user wishes to perform using the device.

[0278] According to one embodiment, the planner module (1425) can generate a plan using the intent and parameters determined by the natural language understanding module (1423). According to one embodiment, the planner module (1425) can determine a plurality of domains necessary to perform a task based on the determined intent. The planner module (1425) can determine a plurality of operations included in each of the plurality of domains determined based on the intent. According to one embodiment, the planner module (1425) can determine parameters necessary to execute the determined plurality of operations or result values ​​output by the execution of the plurality of operations. The parameters and the result values ​​can be defined as concepts related to a specified format (or class). Accordingly, the plan can include a plurality of operations and a plurality of concepts determined by the user's intent. The planner module (1425) can determine the relationships between the plurality of operations and the plurality of concepts in a stepwise (or hierarchical) manner. For example, the planner module (1425) can determine the execution order of a plurality of actions based on the user's intention based on a plurality of concepts. In other words, the planner module (1425) can determine the execution order of a plurality of actions based on parameters required for the execution of the plurality of actions and results output by the execution of the plurality of actions. Accordingly, the planner module (1425) can generate a plan including association information (e.g., ontology) between the plurality of actions and the plurality of concepts. The planner module (1425) can generate the plan using information stored in a capsule database (1430) in which a set of relationships between concepts and actions is stored.

[0279] According to one embodiment, the natural language generation module (1427) can convert specified information into text format. The information converted into text format may be in the form of natural language speech. The text-to-speech conversion module (1429) of one embodiment can convert information in text format into information in speech format.

[0280] According to one embodiment, the capsule database (1430) can store information about the relationship between a plurality of concepts and actions corresponding to a plurality of domains. For example, the capsule database (1430) can store a plurality of capsules including a plurality of action objects (or action information) and concept objects (or concept information) of a plan. According to one embodiment, the capsule database (1430) can store the plurality of capsules in the form of a concept action network (CAN). According to one embodiment, the plurality of capsules can be stored in a function registry included in the capsule database (1430).

[0281] According to one embodiment, the capsule database (1430) may include a strategy registry that stores strategy information required when determining a plan corresponding to a voice input. The strategy information may include reference information for determining a single plan when there are multiple plans corresponding to a user input. According to one embodiment, the capsule database (1430) may include a follow-up registry that stores information on follow-up actions for suggesting follow-up actions to a user in a given situation. The follow-up actions may include, for example, follow-up utterances. According to one embodiment, the capsule database (1430) may include a layout registry that stores layout information of information output through the user terminal (1300). According to one embodiment, the capsule database (1430) may include a vocabulary registry that stores vocabulary information included in capsule information. According to one embodiment, the capsule database (1430) may include a dialog registry in which information about a dialog (or interaction) with a user is stored.

[0282] According to one embodiment, the capsule database (1430) can update stored objects through a developer tool. The developer tool may include, for example, a function editor for updating action objects or concept objects. The developer tool may include a vocabulary editor for updating vocabulary. The developer tool may include a strategy editor for creating and registering strategies that determine plans. The developer tool may include a dialog editor for creating a dialogue with a user. The developer tool may include a follow-up editor for activating follow-up goals and editing follow-up utterances that provide hints. The follow-up goals may be determined based on currently set goals, user preferences, or environmental conditions.

[0283] According to one embodiment, the capsule database (1430) may also be implemented within the user terminal (1300). In other words, the user terminal (1300) may include a capsule database (1430) that stores information for determining an action corresponding to a voice input.

[0284] According to one embodiment, the execution engine (1440) can produce a result using the generated plan. According to one embodiment, the end user interface (1450) can transmit the produced result to the user terminal (1300). Accordingly, the user terminal (1300) can receive the result and provide the received result to the user. According to one embodiment, the management platform (1460) can manage information used in the intelligent server (1400). According to one embodiment, the big data platform (1470) can collect user data. According to one embodiment, the analysis platform (1480) can manage the quality of service (QoS) of the intelligent server (1400). For example, the analysis platform (1480) can manage the components and processing speed (or efficiency) of the intelligent server (1400).

[0285] According to one embodiment, the service server (1500) may provide a designated service (e.g., food ordering or hotel reservation) to the user terminal (1300). According to one embodiment, the service server (1500) may be a server operated by a third party. For example, the service server (1500) may include a first service server (1501), a second service server (1503), and a third service server (1505) operated by different third parties. According to one embodiment, the service server (1500) may provide information for generating a plan corresponding to the received voice input to the intelligent server (1400). The provided information may be stored, for example, in a capsule database (1430). In addition, the service server (1500) may provide result information according to the plan to the intelligent server (1400).

[0286] In the integrated intelligence system (10) described above, the user terminal (1300) can provide various intelligent services to the user in response to user input. The user input may include, for example, input via a physical button, touch input, or voice input.

[0287] According to one embodiment, the user terminal (1300) may provide a voice recognition service through an intelligent app (or voice recognition app) stored internally. In this case, for example, the user terminal (1300) may recognize a user utterance or voice input received through the microphone and provide the user with a service corresponding to the recognized voice input.

[0288] According to one embodiment, the user terminal (1300) may perform a designated action based on the received voice input, either alone or in conjunction with the intelligent server and / or service server. For example, the user terminal (1300) may execute an app corresponding to the received voice input and perform a designated action through the executed app.

[0289] According to one embodiment, when a user terminal (1300) provides a service together with an intelligent server (1400) and / or a service server, the user terminal may detect user speech using the microphone (1320) and generate a signal (or voice data) corresponding to the detected user speech. The user terminal may transmit the voice data to the intelligent server (1400) using a communication interface (1310).

[0290] According to one embodiment, the intelligent server (1400) may generate a plan for performing a task corresponding to the voice input received from the user terminal (1300), or a result of performing an operation according to the plan. The plan may include, for example, a plurality of operations for performing a task corresponding to the user's voice input, and a plurality of concepts related to the plurality of operations. The concept may define parameters input to the execution of the plurality of operations, or result values ​​output by the execution of the plurality of operations. The plan may include association information between the plurality of operations and the plurality of concepts.

[0291] In one embodiment, the user terminal (1300) can receive the response using the communication interface (1310). The user terminal (1300) can output a voice signal generated within the user terminal (1300) to the outside using the speaker (1330), or can output an image generated within the user terminal (1300) to the outside using the display (1340).

[0292] FIG. 14 is a diagram showing a form in which relationship information between concepts and actions is stored in a database according to various embodiments.

[0293] The capsule database (e.g., the capsule database (1430) of FIG. 13) of the intelligent server (e.g., the intelligent server (1400) of FIG. 13) may store multiple capsules in the form of a CAN (concept action network) (1600). The capsule database may store an action for processing a task corresponding to a user's voice input and parameters required for the action in the form of a CAN (concept action network). The CAN may represent an organic relationship between an action and a concept that defines parameters required to perform the action.

[0294] The capsule database may store a plurality of capsules (e.g., Capsule A (1601), Capsule B (1604)) corresponding to each of a plurality of domains (e.g., applications). According to one embodiment, one capsule (e.g., Capsule A (1601)) may correspond to one domain (e.g., application). In addition, one capsule may correspond to at least one service provider (e.g., CP 1 (1602), CP 2 (1603), CP 3 (1606), or CP 4 (1605)) for performing a function of a domain related to the capsule. According to one embodiment, one capsule may include at least one operation (1610) and at least one concept (1620) for performing a specified function.

[0295] According to one embodiment, a natural language platform (e.g., the natural language platform (1420) of FIG. 13) can generate a plan for performing a task corresponding to a received speech input using capsules stored in a capsule database. For example, a planner module of the natural language platform (e.g., the planner module (1425) of FIG. 13) can generate a plan using capsules stored in a capsule database. For example, a plan (1607) can be generated using actions (1711, 1713) and concepts (1712, 1714) of Capsule A (1601) and actions (1741) and concepts (1742) of Capsule B (1604).

[0296] FIG. 15 is a diagram showing a screen for processing voice input received through an intelligent app by a user terminal according to various embodiments.

[0297] The user terminal (1300) can execute an intelligent app to process user input through an intelligent server (e.g., the intelligent server (1400) of FIG. 13).

[0298] According to one embodiment, in the 2010 screen, when the user terminal (1300) recognizes a designated voice input (e.g., wake up!) or receives an input via a hardware key (e.g., a dedicated hardware key), the user terminal (1300) may execute an intelligent app for processing the voice input. For example, the user terminal (1300) may execute an intelligent app while the schedule app is running. According to one embodiment, the user terminal (1300) may display an object (e.g., an icon) (1511) corresponding to the intelligent app on a display (e.g., the display (1340) of FIG. 13). According to one embodiment, the user terminal (1300) may receive a voice input by a user's speech. For example, the user terminal (1300) may receive a voice input such as "Tell me my schedule for this week!" According to one embodiment, the user terminal (1300) may display a user interface (UI) (1513) (e.g., an input window) of an intelligent app on which text data of a received voice input is displayed.

[0299] According to one embodiment, on screen 1220, the user terminal (1300) may display a result corresponding to the received voice input on the display. For example, the user terminal (1300) may receive a plan corresponding to the received user input and display "This Week's Schedule" on the display according to the plan.

[0300] Some of the operations described above may be executed (or performed) by an AI (artificial intelligence) system as described with reference to FIG. 16.

[0301] Figure 16 is a schematic diagram of an exemplary AI system.

[0302] Referring to FIG. 16, the AI ​​system (1600) may include an input / output interface (1610), an AI framework (1620), a generative AI model (1630), and / or a knowledge repository (1690).

[0303] The input / output interface (1610) can receive input. The input can include user input and / or data acquired or generated by the electronic device. The data can include images, videos, and / or sensor data generated by at least one processor of the electronic device (e.g., at least one processor (160) or processor (1720)) (e.g., illuminance data around the electronic device acquired from a sensor or sensor hub (e.g., auxiliary processor (1723), posture data (or orientation data) of the electronic device, temperature inside the electronic device (e.g., temperature of the display (220) or temperature of the at least one processor (160)), size information of a display area of ​​the display (220), and / or images acquired through an image sensor of the electronic device (e.g., included in a camera module (1780)). The user input may include natural language, touch data obtained through touch circuitry included within the display panel (160) (e.g., used to identify input from a finger and / or a stylus), images displayed (and / or to be displayed) on the display panel (160), and / or video. As a non-limiting example, the user input may be received by the input / output interface (1610) together with context information. The context information may be described as additional information obtained in relation to the user input. The context information may relate to a state when the user input is received (e.g., including a state of the electronic device and / or a state surrounding the electronic device (e.g., a user state)). For example, the context information may include information about one or more software applications running within the electronic device when the user input is received. For example, the context information may include information about a location of the electronic device (or a location of a user of the electronic device) when the user input is received.For example, the user input may be integrated with the contextual information. For example, the user input integrated with the contextual information may be received by the input / output interface (1610).

[0304] The input / output interface (1610) can transmit (or provide) output. The output may include a result (or result information) generated or obtained by the AI ​​system (1600) based at least in part on the input. The format of the output may vary. For example, the output may include natural language. For example, the output may include content (e.g., including media content and / or multimedia content). For example, the output may include an action related to a user of the electronic device. For example, the output may have a format according to a user setting of the electronic device.

[0305] The input / output interface (1610) can be described as a user query / response interface (1610).

[0306] The AI ​​framework (1620) can be used to obtain information (or data) about the input from the input / output interface (1610) and control one or more components related to the AI ​​system (1600) using the obtained information.

[0307] For example, the prompt design component (1621) within the AI ​​framework (1620) can use the acquired information to generate or obtain prompts for a generative AI model (1630) (e.g., including a large language model (LLM) or a large multimodal model (LMM)). For example, the prompt design component (1621) can be described as an AI component that utilizes a learning algorithm and / or a neural network to provide enhanced prompts over time. For example, the prompt design component (1621) can use the acquired information to access a knowledge component (e.g., a knowledge repository (1690)) that includes user preference data, a prompt library, and / or prompt examples to generate or obtain prompts. The generated prompts can be provided to the generative AI model (1630) (e.g., including an LLM or LMM).

[0308] For example, the API / plugin management component (1622) within the AI ​​framework (1620) may be utilized to facilitate communication for additional information requested (or induced) in connection with the prompt provided (or to be provided) to the generative AI model (1630). For example, the API / plugin management component (1622) may be utilized to create or establish channels for communication with various data sources (e.g., knowledge repositories (1690)). For example, the API / plugin management component (1622) may facilitate access to at least some of the data sources. For example, the API / plugin management component (1622) may be utilized to request another component (e.g., an application / service component (1680)) to perform feedback (or response) in response to the prompt. As a non-limiting example, information obtained (or generated) through the API / plugin management component (1622) may be provided to the prompt design component (1621) for generating a prompt. As a non-limiting example, information obtained (or generated) through the API / plugin management component (1622) may be provided to a generative AI model (1630).

[0309] For example, the improvement component (1623) within the AI ​​framework (1620) can at least partially tune (or adjust) (or change) the result (e.g., content) obtained (or output) from the generative AI model (1630). For example, the improvement component (1623) can determine or verify whether the content obtained from the generative AI model (1630) is relevant to the input. For example, the improvement component (1623) can determine or verify whether the content obtained from the generative AI model (1630) contains biased content. For example, the improvement component (1623) can determine or verify whether the content obtained from the generative AI model (1630) contains harmful content. For example, the improvement component (1623) can support or assist in performing additional processing to improve the content obtained from the generative AI model (1630). For example, the improvement component (1623) may support providing hints to the user to improve the content.

[0310] A generative AI model (1630) can be described as an artificial intelligence neural network that generates feedback in response to a prompt. For example, the feedback may include additional data and / or information related to the prompt, but relative to the prompt. For example, the feedback may include new content related to the prompt. For example, the generative AI model (1630) may include a model that generates images and / or a model that generates language. For example, the model that generates images may include a generative adversarial network (GAN) and / or a variational autoencoder (VAE). For example, the model that generates images may include a diffusion-based generative model (e.g., a transformer VAE). For example, the model that generates language may include CHAT-GPT 3 and / or CHAT-GPT 4. For example, a generative AI model (1630) may include an LMM that generates the feedback by recognizing text, images, and / or speech.

[0311] As a non-limiting example, the AI ​​framework (1620) and / or the generative AI model (1630) may be included within an AI module (e.g., including a processing circuit) within the electronic device. For example, the AI ​​module may be operatively coupled with at least one processor of the electronic device. For example, the AI ​​module may be operatively coupled with a display driving circuit of the electronic device. For example, the AI ​​module may be operatively coupled with a sensor hub of the electronic device for one or more sensors within the electronic device.

[0312] According to one embodiment, an electronic device may include at least one processor including communication circuitry, a memory including instructions and one or more storage media, and a processing circuit. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to receive a language-based request from a user of the electronic device, transmit input data including the language-based request to a server including a first trained model via the communication circuitry, receive information including an intent included in the language-based request from the server via the communication circuitry, search resources available within the electronic device to obtain a parameter associated with the intent in the information, identify at least one candidate parameter among a plurality of candidate parameters obtained according to the search as at least one parameter to be provided to the server using a second trained model within the electronic device, the second trained model having a lower complexity than the first trained model, and transmit data including the at least one parameter to the server to obtain output data according to the at least one parameter using the first trained model, receive the output data from the server, and provide a response to the language-based request using the output data.

[0313] According to one embodiment, the information including the intent may include information regarding the type of a parameter used to obtain the output data.

[0314] In one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to identify at least one query to obtain a parameter associated with the intent, and to obtain, based on the at least one query, the plurality of candidate parameters from the resources available within the electronic device.

[0315] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to set a search range for obtaining parameters related to the intent and to search for the resources available within the electronic device within the set search range.

[0316] According to one embodiment, the resources available within the electronic device may include resources obtained based on at least one software or resources obtained based on at least one hardware.

[0317] In one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to divide the plurality of candidate parameters into one or more parameter sets based on a reference size processable by the second trained model, sequentially input each of the one or more parameter sets into the second trained model, and identify the at least one candidate parameter based on outputs of the second trained model.

[0318] In one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to remove, from the at least one candidate parameter, a parameter relating to restricted personal information of the user.

[0319] In one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to provide candidate responses as a response to the language-based request using the output data.

[0320] According to one embodiment, the output data may be generated based on the first training model into which the data including the input data and the at least one parameter is input.

[0321] According to one embodiment, a method performed by an electronic device may include receiving a language-based request from a user of the electronic device, transmitting input data including the language-based request to a server including a first trained model through the communication circuit, receiving information including an intent included in the language-based request from the server through the communication circuit, searching resources available within the electronic device to obtain a parameter related to the intent included in the information, using a second trained model within the electronic device, the second trained model having a lower complexity than the first trained model, to identify at least one candidate parameter among a plurality of candidate parameters obtained according to the search as at least one parameter to be provided to the server, transmitting data including the at least one parameter to the server to obtain output data according to the at least one parameter using the first trained model, receiving the output data from the server, and providing a response to the language-based request using the output data.

[0322] According to one embodiment, the information including the intent may include information regarding the type of a parameter used to obtain the output data.

[0323] According to one embodiment, the method may include the steps of: identifying at least one query to obtain parameters associated with the intent; and, based on the at least one query, obtaining the plurality of candidate parameters from the resources available within the electronic device.

[0324] According to one embodiment, the method may include an operation of setting a search range for obtaining parameters related to the intent, and an operation of searching for the resources available within the electronic device within the set search range.

[0325] According to one embodiment, the resources available within the electronic device may include resources obtained based on at least one software or resources obtained based on at least one hardware.

[0326] According to one embodiment, the method may include dividing the plurality of candidate parameters into one or more parameter sets based on a reference size that is processable by the second trained model, sequentially inputting each of the one or more parameter sets into the second trained model, and identifying the at least one candidate parameter based on outputs of the second trained model.

[0327] In one embodiment, the method may include removing, from the at least one candidate parameter, a parameter relating to restricted personal information of the user.

[0328] In one embodiment, the method may include providing candidate responses as a response to the language-based request using the output data.

[0329] According to one embodiment, the output data may be generated based on the first training model into which the data including the input data and the at least one parameter is input.

[0330] According to one embodiment, a non-transitory computer-readable storage medium can store one or more programs. The one or more programs may include instructions that, when executed by at least one processor of an electronic device including a communication circuit, cause the electronic device to receive a language-based request from a user of the electronic device, transmit input data including the language-based request to a server including a first trained model via the communication circuit, receive information including an intent included in the language-based request from the server via the communication circuit, search resources available within the electronic device to obtain a parameter related to the intent in the information, identify at least one candidate parameter among a plurality of candidate parameters obtained according to the search as at least one parameter to be provided to the server using a second trained model within the electronic device, the second trained model having a lower complexity than the first trained model, and transmit data including the at least one parameter to the server to obtain output data according to the at least one parameter using the first trained model, receive the output data from the server, and provide a response to the language-based request using the output data.

[0331] According to one embodiment, the one or more programs may include instructions that, when executed by the at least one processor, cause the electronic device to identify at least one query to obtain a parameter associated with the intent, and to obtain, based on the at least one query, the plurality of candidate parameters from the resources available within the electronic device.

[0332] According to one embodiment, an electronic device may include a communication circuit, a memory including instructions and one or more storage media, and at least one processor including a processing circuit. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to receive a language-based request from a user of the electronic device, transmit input data including the language-based request to a server including a trained model through the communication circuit, receive information including an intent included in the language-based request from the server through the communication circuit, obtain data related to the intent based on data obtained through a plurality of applications of the electronic device, obtain output data corresponding to the input data according to the data related to the intent, transmit the data related to the intent to the server, receive the output data from the server, and provide a response to the language-based request using the output data.

[0333] For example, the information including the intent may include information indicating the type of data associated with the intent used to obtain the output data.

[0334] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to identify at least one query to obtain the data related to the intent, and, based on the at least one query, obtain the data related to the intent from among the data obtained through the plurality of applications.

[0335] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to obtain the data related to the intent based on setting a search range for obtaining the data related to the intent and searching for the data obtained through the plurality of applications within the set search range.

[0336] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to obtain candidate data based on searching the data acquired through the plurality of applications within the set search range, and to obtain the data related to the intent among the candidate data using another trained model having a size smaller than the size of the trained model.

[0337] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to divide the candidate data into data having a reference size processable by the other trained model, sequentially input the candidate data divided into data having the reference size into the other trained model, and obtain the data related to the intent based on outputs of the other trained model.

[0338] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to remove data relating to restricted personal information of the user from the data associated with the intent.

[0339] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to use the output data to provide candidate responses as a response to the language-based request.

[0340] For example, the output data may be generated based on the training model into which the data including the input data and the data regarding the intent has been input.

[0341] According to one embodiment, a method performed by an electronic device may include receiving a language-based request from a user of the electronic device, transmitting input data including the language-based request to a server including a trained model through the communication circuit, receiving information including an intent included in the language-based request from the server through the communication circuit, obtaining data related to the intent based on data obtained through a plurality of applications of the electronic device, transmitting the data related to the intent to the server to obtain output data corresponding to the input data according to the data related to the intent, receiving the output data from the server, and providing a response to the language-based request using the output data.

[0342] For example, the information including the intent may include information indicating the type of data associated with the intent used to obtain the output data.

[0343] For example, the method may include an operation of identifying at least one query to obtain the data related to the intent, and an operation of obtaining the data related to the intent from among the data obtained through the plurality of applications based on the at least one query.

[0344] For example, the method may include an operation of setting a search range for obtaining the data related to the intent, and an operation of obtaining the data related to the intent based on searching the data obtained through the plurality of applications within the set search range.

[0345] For example, the method may include an operation of obtaining candidate data based on searching the data obtained through the plurality of applications within the set search range, and an operation of obtaining the data related to the intent among the candidate data using another trained model having a size smaller than the size of the trained model.

[0346] For example, the method may include an operation of dividing the candidate data into data having a reference size that can be processed by the other trained model, an operation of sequentially inputting the candidate data divided into data having the reference size into the other trained model, and an operation of obtaining the data related to the intent based on outputs of the other trained model.

[0347] For example, the method may include an action of removing data regarding restricted personal information of the user from the data associated with the intent.

[0348] For example, the method may include providing candidate responses as a response to the language-based request using the output data.

[0349] For example, the output data may be generated based on the first training model into which the data including the input data and the data regarding the intent has been input.

[0350] According to one embodiment, a non-transitory computer-readable storage medium may store one or more programs. The one or more programs may include instructions that, when executed by at least one processor of an electronic device including communication circuitry, cause the electronic device to receive a language-based request from a user of the electronic device, transmit input data including the language-based request to a server including a trained model through the communication circuitry, receive information including an intent included in the language-based request from the server through the communication circuitry, obtain data associated with the intent based on data acquired through a plurality of applications of the electronic device, obtain output data corresponding to the input data according to the data associated with the intent, transmit the data associated with the intent to the server, receive the output data from the server, and provide a response to the language-based request using the output data.

[0351] For example, the one or more programs may include instructions that, when executed by the at least one processor, cause the electronic device to identify at least one query to obtain the data related to the intent, and, based on the at least one query, obtain the data related to the intent from among data obtained through the plurality of applications.

[0352] According to one embodiment, an electronic device may include communication circuitry, a memory including instructions and one or more storage media, and at least one processor including processing circuitry. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to receive a language-based request from a user of the electronic device, transmit input data including the language-based request to a server including a trained model through the communication circuitry, receive from the server through the communication circuitry first output data corresponding to the input data and information including an intent included in the language-based request, obtain data related to the intent based on data obtained through a plurality of applications of the electronic device, obtain second output data based on the data related to the intent and the first output data, and provide a response to the language-based request using the second output data.

[0353] For example, in the 21st paragraph, the information including the intent may include information for indicating the type of data related to the intent used to obtain the second output data.

[0354] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to set a search range for obtaining the data related to the intent, and to obtain the data related to the intent based on searching the data obtained through the plurality of applications within the set search range.

[0355] According to one embodiment, an electronic device may include at least one processor including communication circuitry, a memory including instructions and one or more storage media, and a processing circuit. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to receive a language-based request from a user of the electronic device, transmit first data regarding the language-based request to a server including a trained model through the communication circuit, receive second data generated based on the trained model and corresponding to the first data from the server through the communication circuit to provide a response corresponding to the language-based request, obtain third data related to the language-based request based on data acquired through a plurality of applications on the electronic device, and transmit information regarding the third data stored in the electronic device to the server to obtain output data based on the third data stored in the electronic device, receive the output data from the server, and provide a response to the language-based request using the output data.

[0356] According to one embodiment, the second data may include information indicating a type of the third data used to obtain the output data.

[0357] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to identify at least one query to obtain the third data, and to obtain the third data from among the data obtained through the plurality of applications based on the at least one query.

[0358] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to obtain the third data based on setting a search range for obtaining the third data and searching for the data obtained through the plurality of applications within the set search range.

[0359] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to obtain candidate data based on searching the data acquired through the plurality of applications within the set search range, and to obtain third data among the candidate data using another trained model having a size smaller than the size of the trained model.

[0360] In one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to divide the candidate data into data having a reference size processable by the other trained model, sequentially input the candidate data divided into data having the reference size into the other trained model, and obtain the third data based on outputs of the other trained model.

[0361] In one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to remove data relating to restricted personal information of the user from the third data.

[0362] In one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to provide candidate responses as a response to the language-based request using the output data.

[0363] According to one embodiment, the output data may be generated based on the training model into which the input data and the third data are input.

[0364] According to one embodiment, a method performed by an electronic device may include receiving a language-based request from a user of the electronic device, transmitting, through the communication circuit, first data regarding the language-based request to a server including a trained model, receiving, through the communication circuit, from the server, second data generated based on the trained model and corresponding to the first data to provide a response corresponding to the language-based request, obtaining, based on data acquired through a plurality of applications of the electronic device, third data stored in the electronic device and related to the language-based request according to the second data, transmitting information regarding the third data stored in the electronic device to the server to obtain output data based on the third data stored in the electronic device, receiving, from the server, the output data, and providing a response to the language-based request using the output data.

[0365] According to one embodiment, the second data may include information indicating a type of the third data used to obtain the output data.

[0366] According to one embodiment, the method may include an operation of identifying at least one query to obtain the third data, and an operation of obtaining the third data from among the data obtained through the plurality of applications based on the at least one query.

[0367] According to one embodiment, the method may include an operation of setting a search range for obtaining the third data, and an operation of obtaining the third data based on searching the data obtained through the plurality of applications within the set search range.

[0368] According to one embodiment, the method may include an operation of obtaining candidate data based on searching the data obtained through the plurality of applications within the set search range, and an operation of obtaining third data among the candidate data using another trained model having a size smaller than the size of the trained model.

[0369] According to one embodiment, the method may include an operation of dividing the candidate data into data having a reference size that can be processed by the other trained model, an operation of sequentially inputting the candidate data divided into data having the reference size into the other trained model, and an operation of obtaining the third data based on outputs of the other trained model.

[0370] In one embodiment, the method may include removing data relating to restricted personal information of the user from the third data.

[0371] In one embodiment, the method may include providing candidate responses as a response to the language-based request using the output data.

[0372] According to one embodiment, the output data may be generated based on the first training model into which the input data and the third data are input.

[0373] According to one embodiment, a non-transitory computer-readable storage medium can store one or more programs. The one or more programs may include instructions that, when executed by at least one processor of an electronic device including a communication circuit, cause the electronic device to receive a language-based request from a user of the electronic device, transmit first data regarding the language-based request to a server including a trained model through the communication circuit, receive second data generated based on the trained model and corresponding to the first data from the server through the communication circuit to provide a response corresponding to the language-based request, obtain third data related to the language-based request based on data acquired through a plurality of applications of the electronic device, and transmit information regarding the third data stored in the electronic device to the server to obtain output data based on the third data stored in the electronic device, receive the output data from the server, and use the output data to cause the electronic device to provide a response to the language-based request.

[0374] According to one embodiment, the one or more programs may include instructions that, when executed by the at least one processor, cause the electronic device to identify at least one query to obtain the third data, and to obtain the third data from among the data obtained through the plurality of applications based on the at least one query.

[0375] According to one embodiment, an electronic device may include a communication circuit, a memory including instructions and one or more storage media, and at least one processor including a processing circuit. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to receive a language-based request from a user of the electronic device, transmit first data regarding the language-based request to a server including a trained model through the communication circuit, receive first output data generated based on the trained model and second data corresponding to the first data from the server through the communication circuit to provide a response corresponding to the language-based request, obtain third data stored in the electronic device based on the second data based on data obtained through a plurality of applications of the electronic device, obtain second output data based on the third data and the first output data, and provide a response to the language-based request using the second output data.

[0376] According to one embodiment, the second data may include information indicating a type of the third data used to obtain the second output data.

[0377] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to obtain the third data based on setting a search range for obtaining the third data and searching for the data obtained through the plurality of applications within the set search range.

[0378] Electronic devices according to embodiments disclosed herein may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to embodiments disclosed herein are not limited to the aforementioned devices.

[0379] The embodiments of this document and the terminology used herein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among the phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another component (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0380] In one embodiment of this document, the term "module" used may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0381] One embodiment of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0382] According to one embodiment, the method according to one embodiment disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a compact disc read only memory (CD-ROM)), or may be distributed online (e.g., by download or upload) through an application store (e.g., Play Store) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0383] According to one embodiment, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and arranged in other components. According to one embodiment, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to one embodiment, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.< / location> < / location> < / date> < / time> < / location> < / location>

Claims

1. In electronic devices, communication circuit; A memory including instructions and one or more storage media; and At least one processor comprising a processing circuit, The above instructions, when individually or collectively executed by the at least one processor, Receiving a language-based request from a user of the electronic device, Through the above communication circuit, first data regarding the language-based request is transmitted to a server including a trained model, Through the communication circuit, from the server, second data corresponding to the first data is received, which is generated based on the trained model, to provide a response corresponding to the language-based request; Based on data acquired through multiple applications of the electronic device, third data stored in the electronic device and related to the language-based request is acquired according to the second data, Transmitting information about the third data stored in the electronic device to the server to obtain output data based on the third data stored in the electronic device, Receive the output data from the above server, Causing the electronic device to provide a response to the language-based request using the output data; Electronic devices.

2. In the first paragraph, the second data is, Including information for indicating the type of the third data used to obtain the output data, Electronic devices.

3. In the first paragraph, when the instructions are individually or collectively executed by the at least one processor, To obtain the above third data, at least one query is identified, Causing the electronic device to obtain the third data among the data obtained through the plurality of applications based on the at least one query. Electronic devices.

4. In the third paragraph, when the instructions are individually or collectively executed by the at least one processor, Set the search range to obtain the above third data, Causing the electronic device to acquire the third data based on searching the data acquired through the plurality of applications within the set search range. Electronic devices.

5. In the fourth paragraph, when the instructions are individually or collectively executed by the at least one processor, Based on searching the data acquired through the multiple applications within the above-described search range, candidate data is acquired, Causing the electronic device to obtain the third data among the candidate data by using another trained model having a size smaller than the size of the trained model. Electronic devices.

6. In the fifth paragraph, when the instructions are individually or collectively executed by the at least one processor, Divide the above candidate data into data having a standard size that can be processed by the other trained model, The candidate data classified as data having the above reference size are sequentially input into the other trained model, Causing the electronic device to obtain the third data based on the outputs of the other trained model; Electronic devices.

7. In the first paragraph, when the instructions are individually or collectively executed by the at least one processor, Causing the electronic device to remove data regarding restricted personal information of the user from the third data; Electronic devices.

8. In the first paragraph, when the instructions are individually or collectively executed by the at least one processor, Causing the electronic device to provide candidate responses as a response to the language-based request using the output data; Electronic devices.

9. In the first paragraph, the output data is, Based on the training model into which the above input data and the third data are input, Electronic devices.

10. In a method performed by an electronic device, An action of receiving a language-based request from a user of the electronic device; An operation of transmitting first data regarding the language-based request to a server including a trained model through the communication circuit; An operation of receiving, through the communication circuit, second data generated based on the trained model and corresponding to the first data, from the server, to provide a response corresponding to the language-based request; An operation of acquiring third data related to the language-based request and stored in the electronic device based on data acquired through multiple applications of the electronic device, according to the second data; An action of transmitting information about the third data stored in the electronic device to the server to obtain output data based on the third data stored in the electronic device; An operation of receiving the output data from the server; and An operation comprising providing a response to the language-based request using the above output data, method.

11. In the 10th paragraph, the second data is, Including information for indicating the type of the third data used to obtain the output data, method.

12. In the 10th paragraph, the method, To obtain the third data, an operation of identifying at least one query; and An operation of acquiring the third data among the data acquired through the plurality of applications based on at least one query, method.

13. In the 12th paragraph, the method, An operation for setting a search range for obtaining the third data; and An operation of acquiring the third data based on searching the data acquired through the plurality of applications within the set search range, method.

14. In the 13th paragraph, the method, An operation of acquiring candidate data based on searching the data acquired through the plurality of applications within the above-described set search range; and An operation of obtaining the third data among the candidate data by using another trained model having a size smaller than the size of the trained model, method.

15. In a non-transitory computer-readable storage medium storing one or more programs, the one or more programs, when executed by at least one processor of an electronic device including a communication circuit, Receiving a language-based request from a user of the electronic device, Through the above communication circuit, first data regarding the language-based request is transmitted to a server including a trained model, Through the communication circuit, from the server, second data corresponding to the first data is received, which is generated based on the trained model, to provide a response corresponding to the language-based request; Based on the data obtained through the plurality of applications of the electronic device, third data stored in the electronic device and related to the language-based request is obtained according to the second data, Transmitting information about the third data stored in the electronic device to the server to obtain output data based on the third data stored in the electronic device, Receive the output data from the above server, Instructions for causing the electronic device to provide a response to the language-based request using the output data, Non-transitory computer-readable storage medium.

Images