Electronic device for providing response to user input on basis of usage pattern and operation method thereof

Abstract

The present disclosure provides a method for an electronic device. The method may comprise the operations of: acquiring user input data calling an assistant application of the electronic device; acquiring, through a prediction model, pattern information associated with use of the assistant application, on the basis of the user input data and usage history data; generating a prompt on the basis of the pattern information, wherein the prompt comprises first prompt information comprising at least one of a prediction query and a prediction request acquired on the basis of the pattern information; generating a response to the prompt by using a generative AI model; and providing a message comprising the response.

Description

Electronic device and method of operating the same for providing a response to user input based on usage patterns

[0001] The present disclosure relates to an electronic device and an operating method that provides a response to a user input based on a usage pattern.

[0002] With the development of digital technology, electronic devices are available in various forms, such as smart phones, tablet personal computers, or PDAs (personal digital assistants).

[0003] Electronic devices can provide assistant services for user convenience. Assistant services are services that help users perform various tasks through user commands. Applications for these assistant services can be installed on various electronic devices, enhancing convenience in daily life.

[0004] The above information may be provided as background art to aid in understanding the present disclosure. No claim or determination is made as to whether any of the above-described matters constitute prior art related to the present disclosure.

[0005] According to one embodiment, an electronic device may be provided. The electronic device may include at least one processor including a processing circuit and a memory including at least one storage medium storing instructions. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to perform at least one operation. The at least one operation may include obtaining user input data for invoking an assistant application of the electronic device. The at least one operation may include obtaining pattern information associated with the use of the assistant application through a predictive model based on the user input data and usage history data. The at least one operation may include generating a prompt based on the pattern information. The prompt may include first prompt information including at least one of a predictive query or a predictive request obtained based on the pattern information. The at least one operation may include generating a response to the prompt using an artificial intelligence (AI) model. The at least one operation may include providing a message including the response.

[0006] According to one embodiment, a method of operating an electronic device may be provided. The method of operating an electronic device may include at least one operation. The at least one operation may include an operation of obtaining user input data for invoking an assistant application of the electronic device. The at least one operation may include an operation of obtaining pattern information associated with the use of the assistant application through a predictive model based on the user input data and usage history data. The at least one operation may include an operation of generating a prompt based on the pattern information. The prompt may include first prompt information including at least one of a predictive query or a predictive request obtained based on the pattern information. The at least one operation may include an operation of generating a response to the prompt using an artificial intelligence (AI) model. The at least one operation may include an operation of providing a message including the response.

[0007] According to one embodiment, a storage medium storing at least one computer-readable instruction may be provided. The at least one instruction, when executed by at least a portion of at least one processor of an electronic device, may cause the electronic device to perform at least one operation. The at least one operation may include obtaining user input data for invoking an assistant application of the electronic device. The at least one operation may include obtaining pattern information associated with the use of the assistant application through a predictive model based on the user input data and usage history data. The at least one operation may include generating a prompt based on the pattern information. The prompt may include first prompt information including at least one of a predictive query or a predictive request obtained based on the pattern information. The at least one operation may include generating a response to the prompt using an artificial intelligence (AI) model. The at least one operation may include providing a message including the response.

[0008] In connection with the description of the drawings, the same or similar reference numerals may be used for identical or similar components.

[0009] FIG. 1 is a block diagram of an electronic device within a network environment according to various embodiments of the present disclosure.

[0010] FIG. 2 is a diagram illustrating a generative artificial intelligence system according to one embodiment of the present disclosure.

[0011] FIG. 3 is a diagram illustrating a configuration of a first electronic device according to one embodiment of the present disclosure.

[0012] FIG. 4 is a diagram illustrating a configuration of a second electronic device according to one embodiment of the present disclosure.

[0013] FIG. 5 is a flowchart illustrating a method for an electronic device to provide a response to a user input through an assistant application, according to one embodiment of the present disclosure.

[0014] FIG. 6 is a diagram illustrating an operation of an electronic device providing a response to a user input through an assistant application using a server according to one embodiment of the present disclosure.

[0015] FIG. 7 is a diagram illustrating an operation of an electronic device providing a response to a user input through an assistant application according to one embodiment of the present disclosure.

[0016] FIG. 8 is a flowchart illustrating an operation of an electronic device collecting at least one piece of relevant information based on usage pattern information, according to one embodiment of the present disclosure.

[0017] FIG. 9 is a diagram illustrating an operation of an electronic device collecting at least one piece of relevant information through a prediction module according to one embodiment of the present disclosure.

[0018] FIG. 10A is a flowchart illustrating an operation of an electronic device generating a prompt according to one embodiment of the present disclosure.

[0019] FIG. 10b is a flowchart illustrating a response providing procedure including a prompt generation operation by an electronic device according to one embodiment of the present disclosure.

[0020] FIG. 10c is a flowchart illustrating a response providing procedure including a prompt generation operation by multiple electronic devices according to one embodiment of the present disclosure.

[0021] FIG. 11 is a diagram illustrating an operation of an electronic device generating a prompt through a prompt generation module according to one embodiment of the present disclosure.

[0022] FIG. 12 is a diagram illustrating a response to a user input according to one embodiment of the present disclosure.

[0023] FIGS. 13A to 13C are diagrams illustrating responses to user input based on analysis of usage patterns of an assistant application according to one embodiment of the present disclosure.

[0024] FIG. 14 is a diagram illustrating a response to a user input according to a usage pattern of an assistant application according to one embodiment of the present disclosure.

[0025] FIG. 15 is a diagram illustrating a method for an electronic device to provide a response using a gesture input, according to one embodiment of the present disclosure.

[0026] FIG. 16 is a diagram illustrating a method for an electronic device to provide a response including visual information, according to one embodiment of the present disclosure.

[0027] FIGS. 17A to 17G are diagrams illustrating responses including visual information according to one embodiment of the present disclosure.

[0028] FIGS. 18A to 18C are diagrams illustrating screens for setting a welcome message according to one embodiment of the present disclosure.

[0029] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. In connection with the description of the drawings, the same or similar reference numerals may be used for identical or similar components. Furthermore, in the drawings and related descriptions, descriptions of well-known functions and configurations may be omitted for clarity and conciseness.

[0030] FIG. 1 is a block diagram of an electronic device within a network environment according to various embodiments of the present disclosure.

[0031] Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with an electronic device (104) or a server (108) via a second network (199) (e.g., a long-range wireless communication network). In one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0032] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or operations. According to one embodiment, as at least a part of the data processing or operations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or an auxiliary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together with the main processor (121). For example, when the electronic device (101) includes the main processor (121) and the auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a given function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0033] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, in the electronic device (101) itself where artificial intelligence is performed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0034] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0035] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0036] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0037] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0038] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0039] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0040] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0041] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0042] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0043] The haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. According to one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0044] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0045] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0046] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0047] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a communication module (192) (e.g., a cellular communication module, a short-range communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0048] The communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The communication module (192) may support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The communication module (192) may support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the communication module (192) may support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0049] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas, for example, by the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device via the at least one selected antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0050] According to various embodiments, the antenna module (197) may form a mmWave antenna module. In one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

[0051] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface).

[0052] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In one embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server using machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0053] The number of processors (120) may be one or more. For example, the processor (120) may have a multi-core processor structure such as a dual core, quad core, or hexa core.

[0054] The processor (120) can control the operations of the electronic device (101) by executing instructions stored in the memory (130). For example, the processor (120) can correspond to a plurality of processors that collectively perform a plurality of operations by dividing them among the processors.

[0055] FIG. 2 is a diagram illustrating a generative artificial intelligence system according to one embodiment of the present disclosure.

[0056] According to one embodiment, a user query / response interface (210) may receive input (e.g., user input or data acquired or generated by an electronic device (e.g., electronic device (101) of FIG. 1). The data acquired or generated by the electronic device may include, for example, image or video data generated using a processor (e.g., processor (120) of FIG. 1), values ​​transmitted through a sensor (e.g., sensor module (176) of FIG. 1) or sensor hub (e.g., external illuminance, angle of the electronic device, temperature of the display (e.g., display module (160) of FIG. 1) or electronic device, display size or expansion / reduction information, captured images of an image sensor). The user input may be in the form of natural language, touch coordinates or stylus coordinates acquired through a touch panel or digitizer included in the display, images and / or videos, but is not limited thereto. In addition, context information may also be transmitted when transmitting the user input. The context information may include various additional information at the time of user input. It can include information. For example, the additional information can include information about the application the user is currently using or the user's location information. In addition, the user input can be a mixed form of the above-described natural language, images, sounds, and context information. In addition, the user input can also be a non-natural language form, such as selecting a menu. The user query / response interface (210) can output the results of the generative artificial intelligence system (200) and / or the results of analyzing the input to the user. The output can be in the form of natural language or a specific content, and can also be provided in the form of an action requested by the user. The user query / response interface (210) can output the results of the generative artificial intelligence system (200) to the user. The output can be in the form of natural language or a specific content, and can also be provided in the form of an action requested by the user.

[0057] According to one embodiment, the AI ​​framework (240) can receive user input and coordinate and control each component necessary to perform the user's intention based on the user's query.

[0058] According to one embodiment, user input received from the user query / response interface (210) may be transmitted to a prompt design component (241). The prompt design component (241) may be used to generate prompts suitable for inputting the user input into a large language model (LLM) or a large multimodal model (LMM). The prompt design component (241) may be an AI component that uses a machine learning algorithm or a neural network to develop better prompts over time. The prompt design component (241) may access a knowledge component including user preference data, a prompt library, and prompt examples based on the user input to generate prompts, and may transmit the generated prompts to the LLM or LMM.

[0059] According to one embodiment, the API / Plug-in management component (242) may communicate with external information when there is a request for additional information when passing user input as input to the generative model. The API / Plug-in management component (242) may establish a channel for communicating with the outside of the AI ​​Interface through the API, and may enable access to various data sources (e.g., knowledge repositories (220)) through the established channel. In addition, the API / Plug-in management component (242) may request the application / service component (230) through the API to perform an action that ultimately performs the user input, rather than an intermediate result, when the action needs to be performed in the application or service. Information obtained from the outside may be used to generate a prompt in the prompt design component (241) together with the user input, or may be passed as an input to the generative model.

[0060] According to one embodiment, the output modification component (also referred to as a refiner component) (243) can fine-tune the output from the generative model. For example, the output modification component (243) can verify that the content generated through the LLM and / or LMM is not irrelevant, does not contain biased content, or does not contain harmful content. In addition, the output modification component (243) can determine to what extent the content matches the result desired by the user and, if necessary, can perform additional processing. The output modification component (243) can additionally configure and provide the user with hints to avoid undesired output.

[0061] According to one embodiment, a generative AI model (260) may generally refer to an artificial intelligence neural network that creates new types of data based on user input information. The generative AI model (260) may include a model that generates images and / or a model that generates languages. Representative models that generate images include a generative adversarial network (GAN) and a variational auto encoder (VAE), and examples include a diffusion-based generative model that uses a VAE and a Transformer structure. A model that generates languages ​​is a model that is trained to statistically output the most appropriate output based on input values, and representative examples include models such as CHAT-GPT 3 and CHAT-GPT 4. In addition, there is also an LMM (large multimodal model) that can recognize various types of data input, such as text, images, and voice, and generate new data corresponding thereto.

[0062] FIG. 3 is a diagram illustrating a configuration of a first electronic device according to one embodiment of the present disclosure.

[0063] FIG. 4 is a diagram illustrating a configuration of a second electronic device according to one embodiment of the present disclosure.

[0064] The components and operations of the components described with reference to FIGS. 3 to 4 may be partially or entirely identical to the components and operations of the components described with reference to FIGS. 1 to 2. The components and operations of the components described with reference to FIGS. 3 to 4 may be partially or entirely identical to the components and operations of the components described with reference to FIGS. 5 to 17g described below.

[0065] According to one embodiment, the first electronic device (300) (e.g., the electronic device (101) of FIG. 1) may be an electronic device including a client-side configuration (e.g., an assistant client) for an intelligent application (e.g., an assistant application). According to one embodiment, the second electronic device (400) (e.g., the server (108) of FIG. 1) may be an electronic device including a server-side configuration (e.g., an assistant server) for an intelligent application (e.g., an assistant application). According to one embodiment, the assistant application may be an application that recognizes and processes a user input (e.g., a user voice input) based on artificial intelligence and performs various tasks based on the user input. The assistant application may be, for example, Samsung's Bixby. TM , Apple's Siri TM , Amazon's Alexa TM This may be, but is not limited to, an assistant application. In this disclosure, the assistant application may be abbreviated as an assistant. In this disclosure, for convenience of explanation, the server is described as an example of the second electronic device (400).

[0066] Referring to FIG. 3, according to one embodiment, the first electronic device (300) may include at least one input interface (310), at least one output interface (320), at least one memory (330), at least one processor (340), at least one communication circuit (350), at least one natural language processing module (360), at least one intelligence module (370), a prediction module (380), and / or a prompt generation module (390).

[0067] According to one embodiment, the input interface (310) can receive user input using an input device (e.g., microphone, touch display) and transmit the received user input to the processor (340).

[0068] According to one embodiment, the output interface (320) can output the results processed by the processor (340) using an output device (e.g., speaker, display).

[0069] According to one embodiment, the memory (330) (e.g., the memory (130) of FIG. 1) may store various data that may be used to control the operation of each component of the electronic device (300). The memory (330) may include, for example, at least one storage medium that stores a plurality of application programs used in the electronic device (300), data for controlling the operation of the electronic device (300), and commands. The commands stored in the memory (340), when executed by at least one processor (340), may cause the electronic device (300) to perform at least one operation (e.g., at least one of the operations of FIGS. 1 to 17e).

[0070] According to one embodiment, the memory (330) can store at least one program for processing and controlling the processor (340), and can store input and / or output data. The memory (330) can also store at least one artificial intelligence model. The memory (330) can include at least one of a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD (secure digital) or XD (extreme digital) memory), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, and an optical disk. According to one example, a web storage or a cloud server that performs a storage function on the Internet may be operated by the electronic device (300).

[0071] According to one embodiment, the processor (340) (e.g., the processor (120) of FIG. 1) may control the overall operation of the electronic device (300) and perform at least one operation of the electronic device (300) (e.g., at least one of the operations of FIGS. 1 to 17g). The processor (340) may perform operations or data processing related to control and / or communication of at least one other component of the electronic device (300). The processor (340) may include at least one processing circuit that executes instructions stored in the memory (330).

[0072] According to one embodiment, at least one processor (340) may include various processing circuits and / or multiple processors. One or more of the at least one processor (340) may be individually and / or collectively configured to perform various functions described herein. In this disclosure, when "a processor," "at least one processor," and "one or more processors" are described as being configured to perform various functions, these terms include, but are not limited to, situations where one processor performs some of the recited functions and other processor(s) perform other parts of the recited functions, and also situations where a single processor can perform all of the recited functions. Additionally, the at least one processor (340) may include a combination of processors that perform the various recited / disclosed functions, for example, in a distributed manner. The at least one processor (340) may execute program instructions to achieve or perform various functions.

[0073] According to one embodiment, at least one processor (340) may include at least one of a central processing unit (CPU), a neural processing unit (NPU), a graphics processing unit (GPU), a micro processing unit (MPU), a micro controller unit (MCU), an application processor (AP), a communication processor (CP), a system on chip (SoC), or an integrated circuit (IC), a sensor hub, a supplementary processor, an application specific integrated circuit (ASIC), or a field programmable gate array (FPGA), and may have multiple cores.

[0074] According to one embodiment, the communication circuit (350) (e.g., the communication module (190) of FIG. 1) may support establishment of a direct (e.g., wired) communication channel or wireless communication channel between the electronic device (300) and an external electronic device, and performance of communication through the established communication channel.

[0075] According to one embodiment, the natural language processing module (360) may include an automatic speech recognition (ASR) module (361), a natural language understanding (NLU) module (362), a natural language generation (NLG) module (363), and / or a text-to-speech (TTS) module (364).

[0076] According to one embodiment, the automatic speech recognition module (361) can convert voice input into text data. The automatic speech recognition module (361) can analyze voice input using a voice database and artificial intelligence algorithms and perform operations to recognize words and sentences.

[0077] According to one embodiment, the natural language understanding module (362) can identify a user's intent and / or parameters based on text data of a voice input. The natural language understanding module (362) can perform syntactic analysis, semantic analysis, contextual understanding, and / or sentiment analysis using natural language processing and artificial intelligence algorithms. The user's intent corresponds to the voice input and may include information indicating an action (or function) that the user wishes to perform using the electronic device.

[0078] In one embodiment, the natural language generation module (363) can convert specified information into text format. The natural language generation module (363) can generate text in a language easily understood by humans using natural language processing and artificial intelligence algorithms. The information converted into text format may be in the form of natural language utterances.

[0079] According to one embodiment, the text-to-speech module (364) can convert information in text form into information in voice form. The text-to-speech module (364) can provide more natural and fluent voice by using an artificial intelligence algorithm.

[0080] According to one embodiment, the intelligent module (370) may include an assistant client (371) and / or a generative AI model (372).

[0081] According to one embodiment, the assistant client (371) may refer to an assistant application running on the electronic device (300). According to one embodiment, the assistant client (371) (or the assistant application) may be activated in response to receiving at least one activation input (or trigger input). The at least one activation input may include, but is not limited to, an input of pressing a specific physical button of the electronic device (300), a voice input of calling a name corresponding to the assistant application (e.g., a voice call), and / or an input of pressing a specific area of ​​the screen of the electronic device (300) (e.g., a long press touch input). According to one embodiment, the assistant client (371) may support obtaining a user input in an activated state and generating a prompt based on the user input.

[0082] In one embodiment, a generative AI model (372) (e.g., the generative AI model (260) of FIG. 2) can generate a response based on a prompt. The generative AI model (372) may include a smaller LLM (sLLM) and / or an LMM. An sLLM is a small-sized language model, for example, a language model that uses a relatively small amount of training data or does not have a large model size itself. An sLMM may have a relatively small number of parameters compared to an LMM. A smaller number of parameters means that the language model is lighter and can execute faster. Therefore, an sLMM may be more suitable than an LMM for some applications. For example, an sLMM may be useful when applied to a limited domain or a specific task.

[0083] According to one embodiment, the prediction module (380) can predict (or analyze) a user's usage pattern and / or a user's speech pattern. For example, the prediction module (380) can predict (or analyze) a user's usage pattern based on the user's usage history of the user's assistant and / or the user's usage history of the user's application. The usage pattern can be used to configure the content of a response. For example, the prediction module (380) can predict (or analyze) a user's speech pattern based on the user's speech history through the user's assistant. The speech pattern can be used to configure the expression method of the response. The prediction module (380) can be implemented as a learned artificial intelligence model. The prediction module (380) can transmit information acquired through the prediction module (380) to the prompt generation module (390).

[0084] According to one embodiment, the prompt generation module (390) may generate a prompt for generating an answer through the generative AI model (372) based on the user's usage pattern and / or the user's speech pattern analyzed by the prediction module. According to one embodiment, the prompt generation module (390) may receive information necessary for prompt generation from the prediction module (380) and reflect the received information in the content of the prompt. According to one embodiment, the prompt generation module (390) may reflect expected content based on the user's usage pattern in the content of the prompt. According to one embodiment, the prompt generation module (390) may reflect content directly requested by the user in the content of the prompt based on the user's speech input.

[0085] Referring to FIG. 4, according to one embodiment, the second electronic device (400) may include at least one memory (410), at least one processor (420), at least one communication circuit (430), at least one natural language processing module (440), at least one intelligence module (450), a prediction module (460), and / or a prompt generation module (470).

[0086] According to one embodiment, the memory (410) (e.g., the memory (130) of FIG. 1 or the memory (330) of FIG. 3) may store various data that may be used to control the operation of each component of the electronic device (400). The memory (410) may include, for example, at least one storage medium that stores a plurality of application programs used in the electronic device (400), data for controlling the operation of the electronic device (400), and commands. The commands stored in the memory (410), when executed by at least one processor (420), may cause the electronic device (400) to perform at least one operation (e.g., at least one of the operations of FIGS. 1 to 17g).

[0087] According to one embodiment, the processor (420) (e.g., the processor (120) of FIG. 1 or the processor (340) of FIG. 3) may control the overall operation of the electronic device (400) and perform at least one operation of the electronic device (400) (e.g., at least one of the operations of FIGS. 1 to 17g). The processor (420) may perform operations or data processing related to control and / or communication of at least one other component of the electronic device (400). The processor (420) may include at least one processing circuit that executes instructions stored in the memory (410).

[0088] According to one embodiment, the communication circuit (430) (e.g., the communication module (190) of FIG. 1) may support establishment of a direct (e.g., wired) communication channel or wireless communication channel between the electronic device (400) and an external electronic device, and performance of communication through the established communication channel.

[0089] According to one embodiment, the natural language processing module (440) may include an automatic speech recognition module (441) (e.g., the automatic speech recognition module (361) of FIG. 3), a natural language understanding module (442) (e.g., the natural language understanding module (362) of FIG. 3), a natural language generation module (443) (e.g., the natural language generation module (363) of FIG. 3), and / or a text-to-speech module (444) (e.g., the text-to-speech module (364) of FIG. 3).

[0090] In one embodiment, the assistant server (451) may be a server-side configuration that is paired with an assistant client (371), which is a client-side configuration. In one embodiment, the assistant server (451) may receive data of user input (e.g., voice input, text input) from the assistant client (371), analyze the user input based on the received data, and generate a response corresponding to the user input or a command for operating a function of the first electronic device (300) and transmit the generated command to the assistant client (371).

[0091] According to one embodiment, a generative AI model (452) (e.g., the generative AI model (260) of FIG. 2) can generate a response based on a prompt. The generative AI model (452) can include an LLM and / or an LMM. An LLM refers to an artificial neural network-based language model that has learned a large amount of text data through pre-training. An LLM can include many more parameters (e.g., more than 10 billion) than a typical language model. An LLM is a type of machine learning model used in the field of natural language processing, and can be used to learn a large amount of text data and make predictions about new text data. An LLM can be utilized for tasks such as natural language understanding, sentence generation, translation, grammatical error correction, and summarization, for example.

[0092] According to one embodiment, the prediction module (460) can predict (or analyze) a user's usage pattern and / or a user's speech pattern. For example, the prediction module (460) can predict (or analyze) a user's usage pattern based on the user's usage history of the user's assistant and / or the user's usage history of the user's application. The usage pattern can be used to configure the content of a response. For example, the prediction module (460) can predict (or analyze) a user's speech pattern based on the user's speech history through the user's assistant. The speech pattern can be used to configure the expression method of the response. The prediction module (460) can be implemented as a learned artificial intelligence model. The prediction module (460) can transmit information acquired through the prediction module (460) to the prompt generation module (470).

[0093] According to one embodiment, the prompt generation module (470) may generate a prompt for generating an answer through the generative AI model (452) based on the user's usage pattern and / or the user's speech pattern analyzed by the prediction module. According to one embodiment, the prompt generation module (470) may receive information necessary for prompt generation from the prediction module (460) and reflect the received information in the content of the prompt. According to one embodiment, the prompt generation module (470) may reflect expected content based on the user's usage pattern in the content of the prompt. According to one embodiment, the prompt generation module (470) may reflect content directly requested by the user in the content of the prompt based on the user's speech input.

[0094] FIG. 5 is a flowchart illustrating a method for an electronic device to provide a response to a user input through an assistant application, according to one embodiment of the present disclosure.

[0095] The components and operations of the components described with reference to FIG. 5 may be partially or entirely identical to the components and operations of the components described with reference to FIGS. 1 to 4. The components and operations of the components described with reference to FIG. 5 may be partially or entirely identical to the components and operations of the components described with reference to FIGS. 6 to 17g, which will be described later.

[0096] In the following examples, the operations may be performed sequentially, but are not necessarily sequential. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0097] According to one embodiment, operations 510 to 540 may be understood to be performed in a processor (e.g., processor (340) of FIG. 3 or processor (420) of FIG. 4) of an electronic device (e.g., first electronic device (300) of FIG. 3 or second electronic device (400) of FIG. 4).

[0098] Referring to FIG. 5, according to one embodiment, in operation 510, an electronic device (e.g., the first electronic device (300) of FIG. 3 or the second electronic device (400) of FIG. 4) may call an assistant application of the electronic device. According to one embodiment, the electronic device may obtain user input data for calling the assistant application of the electronic device. The user input data may include, for example, a voice input, a text input, and / or a gesture input.

[0099] According to one embodiment, the user input data may include a user input for invoking an assistant application. The user input for invoking the assistant application may include, but is not limited to, an input for pressing a specific physical button or virtual button of the electronic device (300) for a specified period of time, a voice input for calling a name corresponding to the assistant application (e.g., a voice call), and / or an input for pressing a specific area of ​​the screen of the electronic device (300) for a specified period of time (e.g., a long press touch input). The voice input may include a portion corresponding to a wake word of the assistant application (e.g., "Hi Bixby") (hereinafter, referred to as the wake word portion or wake portion). The wake word portion may be used to activate the assistant application.

[0100] In one embodiment, user input data (e.g., voice input) may further include a portion (hereinafter, referred to as a user request portion or information portion) corresponding to a user request (e.g., "Search for article information") and / or a user query (e.g., "What's the weather today?"). The user request portion may be used to generate a prompt.

[0101] In one embodiment, the electronic device can distinguish between a wake word portion and a user request portion from user input data. In response to identifying that the wake word portion is included in the user input data, the electronic device can activate an assistant application corresponding to the wake word.

[0102] According to one embodiment, in operation 520, the electronic device may obtain pattern information (hereinafter, “usage pattern information”) and / or at least one piece of related information associated with the use of the assistant application. Operation 530 may be performed by a prediction module (e.g., the prediction module (380) of FIG. 3 or the prediction module (460) of FIG. 4).

[0103] According to one embodiment, the electronic device can obtain usage pattern information through a prediction model (e.g., the prediction model (380) of FIG. 3 or the prediction model (460) of FIG. 4) based on user input data and usage history data.

[0104] In one embodiment, the usage history data may include at least one of information about a usage history of an assistant application and / or information about a usage history of at least one application. The at least one application may be, for example, at least one application used with or without the assistant application (e.g., a weather application, a calendar application). The information about the usage history of the assistant application may include, for example, information about a history of queries, requests, commands, and / or searches made by a user through the assistant application at a specific time and / or at a specific location (e.g., a specific place). The information about the usage history of at least one application may include, for example, information about a history of a user using a function and / or operation of at least one application with or without the assistant application at a specific time and / or at a specific location (e.g., a specific place).

[0105] In one embodiment, the usage pattern information may include information about a user's pattern of using the assistant application and / or information about a pattern of using at least one application. For example, the usage pattern information may include information about a pattern of a user making a query (and / or request) through the assistant application at a specific time and / or location (e.g., a specific place), information about a pattern of a user using a function (and / or operation) of at least one application with or without the assistant application, and / or information about a pattern of a user controlling at least one other electronic device through the assistant application. The usage pattern information may be used to generate a prompt.

[0106] In one embodiment, the predictive model may be an artificial intelligence model (e.g., a machine learning model) trained to obtain usage pattern information based on input data (e.g., user input data and / or usage history data).

[0107] According to one embodiment, an electronic device may collect at least one piece of relevant information through at least one module (e.g., an information collection module) based on usage pattern information. The at least one piece of relevant information may be associated with the usage pattern information. The at least one piece of relevant information may be used to generate a prompt, either together with or separately from the usage pattern information. An example of an operation in which the electronic device collects at least one piece of relevant information is described below with reference to FIGS. 8 and 9 .

[0108] In one embodiment, the electronic device can collect at least one piece of relevant information periodically or upon request. In one embodiment, the electronic device can collect at least one piece of relevant information before or even before the assistant application is activated. This allows the relevant information for generating a prompt to be collected in advance, thereby avoiding delays in collecting relevant information after the assistant application is activated.

[0109] According to one embodiment, in operation 530, the electronic device may generate a prompt based on acquired information (e.g., usage pattern information and / or at least one piece of related information). Operation 530 may be performed by a prompt generation module (e.g., the prompt generation module (390) of FIG. 3 or the prompt generation module (470) of FIG. 4). An example of an operation in which the electronic device generates a prompt is described below with reference to FIGS. 10A to 10C and 11.

[0110] According to one embodiment, the prompt may include first prompt information including at least one of a predicted query (or predicted query) or a predicted request (or predicted request) obtained based on usage pattern information.

[0111] In one embodiment, the prompt may further include expression style information related to the expression style of the response. The expression style of the response may include, but is not limited to, the length of the response, the expression format of the response (e.g., direct expression, indirect expression), acoustic characteristics of the voice corresponding to the response, and / or linguistic characteristics of the response. The expression style information may include, for example, first expression style information regarding the expression style of the first response associated with the first prompt information.

[0112] In one embodiment, the prompt may further include second prompt information comprising at least one user request or user query obtained based on user input data. The presentation method information may further include second presentation method information regarding the presentation method of a second response associated with the second prompt information.

[0113] In one embodiment, the second presentation method information may differ from the first presentation method information. For example, the first presentation method information may include information for indirectly expressing a first response associated with the first prompt information, and the second presentation method information may include information for directly expressing a second response associated with the second prompt information. In this case, even if the first and second responses are responses intended to provide substantially the same content (e.g., a response for providing today's weather information), the first response, which is predicted based on usage patterns, may be expressed indirectly (e.g., "It's a nice sunny day. How can I help you?"), while the second response, which is a response to a direct question or request based on user input, may be expressed directly (e.g., "It's a sunny day with a temperature of 30 degrees."). This method allows for a natural response to be provided to the user. For example, although the predicted first response is based on past usage patterns, it may not be a response that the user actually wants at the present time. Therefore, expressing it directly, like the second response, may cause a sense of rejection to the user. However, if you express it indirectly, you can provide a response to the user without making it seem awkward, even if it is not the response the user actually wants at the moment.

[0114] According to one embodiment, the expression style information (e.g., the first expression style information and / or the second expression style information) may be generated based on at least one piece of speech analysis information obtained by analyzing the content of the user's speech (e.g., speech pattern) through the assistant application. The at least one piece of speech analysis information may include, but is not limited to, for example, the manner of calling the assistant application (speech style or calling method), the time of calling the assistant application (speech time or calling time) (e.g., morning, evening), the location (or place) of calling the assistant application (speech location or calling location) (e.g., home, work), the acoustic characteristics of the user's speech through the assistant application (speech acoustic characteristics) (e.g., calling in a loud, soft, high, low voice), and / or the linguistic characteristics of the user's speech through the assistant application (speech linguistic characteristics) (e.g., speech pattern, tone, sentence length, words used, sentences). Acoustic features of speech may include, but are not limited to, features associated with the physical properties of speech (e.g., waveform, amplitude, frequency, timing, spectrum), for example, and voice. Linguistic features of speech may include, for example, features associated with the structure and meaning of language (e.g., phonetics, vocabulary, grammar, syntax, preferred word and sentence structure, speech style (or tone), sentence length). Expression style information may be set in various forms of information based on at least one piece of speech analysis information. In this way, by determining the expression style of a response by considering various factors, a natural response that suits the user's characteristics, environment, situation, and preferences can be provided to the user.

[0115] According to one embodiment, at operation 540, the electronic device may generate a response to the prompt via a generative AI model (e.g., the generative AI model (372) of FIG. 3 or the generative AI model (452) of FIG. 4) and provide a message including the response.

[0116] In one embodiment, a response to a prompt may include a first response associated with the first prompt information and / or a second response associated with the second prompt information. This allows for personalized, nuanced responses to users by providing responses to queries or requests predicted based on the user's usage patterns, as well as responses to queries or requests directly requested by the user.

[0117] In one embodiment, the message may correspond to a welcome message, the first message received upon invoking (or activating) the assistant application. This allows the welcome message upon invoking the user's assistant application to be personalized and include a detailed response.

[0118] According to one embodiment, a generative AI model (e.g., sLMM, LLM, LMM) can generate a first response associated with the first prompt information based on the first prompt information and the first expression method information. According to one embodiment, the generative AI model can generate a second response associated with the second prompt information based on the second prompt information and the second expression method information. According to one embodiment, when the first expression method information and the second expression method information are different, the first response and the second response can be expressed in different expression methods. For example, the first response may be an indirect expression of a response to a predicted query or a predicted request included in the first prompt information, and the second response may be a direct expression of a response to a user query or a user request included in the second prompt information. In this way, a natural response similar to an actual conversation can be provided to the user.

[0119] FIG. 6 is a diagram illustrating an operation of an electronic device providing a response to a user input through an assistant application using another electronic device, according to one embodiment of the present disclosure.

[0120] The components and operations of the components described with reference to FIG. 6 may be partially or entirely identical to the components and operations of the components described with reference to FIGS. 1 to 5. The components and operations of the components described with reference to FIG. 6 may be partially or entirely identical to the components and operations of the components described with reference to FIGS. 7 to 17g, which will be described later.

[0121] In the embodiment of FIG. 6, it is assumed that the operations by the prediction module (e.g., operation 520 of FIG. 5), the operations by the prompt generation module (e.g., operation 530 of FIG. 5)) and the operations by the generative AI module (e.g., operation 540 of FIG. 5) are performed by a server (e.g., the server (108) of FIG. 1 or the second electronic device (400) of FIG. 4). For example, in the embodiment of FIG. 6, the server may include a prediction module (e.g., the prediction module (460) of FIG. 4) and / or a prompt generation module (e.g., the prompt generation module (470) of FIG. 4), and the electronic device (e.g., the electronic device (101) of FIG. 1 or the first electronic device (300) of FIG. 3) may not include a prediction module (e.g., the prediction module (380) of FIG. 3) and / or a prompt generation module (e.g., the prompt generation module (390) of FIG. 3). When server-based prediction and response operations like this are performed, predictions and responses can be provided by leveraging the server's high computing power and greater resources. However, this may result in delays due to communication between the server and the electronic device.

[0122] In the embodiment of FIG. 6, for convenience of explanation, the user input data for calling the assistant application is described as an example in which the user input data includes a voice input (601) corresponding to the speech of the first user (U1). However, this is not limited thereto, and the same explanation may be applied even when various types of user input (e.g., text input) are used.

[0123] Referring to FIG. 6, according to one embodiment, the electronic device (300) can receive user input data including voice input (601) through the input interface (310). The electronic device (300) can transmit the user input data to the server (400) through the assistant client (371) within the intelligent module (370).

[0124] According to one embodiment, the server (400) may process user input data including a voice input (601) through the ASR module (441) to obtain user input data including text corresponding to the voice input (601). The server (400) may obtain usage pattern information and / or at least one piece of related information using the prediction module (460). The operation of the server (400) obtaining the usage pattern information and / or at least one piece of related information using the prediction module (460) may include operation 520 of FIG. 5 , operations of FIG. 8 , and operations of FIG. 9 . The server (400) may generate a prompt using the prompt generation module (470). The operation of the server (400) generating a prompt using the prompt generation module (470) may include operation 530 of FIG. 5 , operations of FIGS. 10A to 10C , and operations of FIG. 11 . The server (400) can generate a response to a prompt using a generative AI model (452) (e.g., LLM, LMM). The operation of the server (400) generating a response to the prompt using the generative AI model (452) may include operation 540 of FIG. 5 . The server (400) may process a response including text through a TTS module (444) to obtain a response including a voice corresponding to the text. The server (400) may transmit the response to the electronic device (300).

[0125] In one embodiment, the electronic device (300) may transmit the received response to the output interface (320) via the assistant client (371). The electronic device (300) may provide a message (602) including the response via the output interface (320). For example, the electronic device (300) may provide audio included in the response (or message) audibly via a speaker. For example, the electronic device (300) may provide visual information included in the response (or message) visually via a display. According to an embodiment, the electronic device (300) may utilize a generative AI model (372) (e.g., sLMM) of the electronic device (300) to adjust the expression method of the response.

[0126] FIG. 7 is a diagram illustrating an operation of an electronic device providing a response to a user input through an assistant application according to one embodiment of the present disclosure.

[0127] The components and operations of the components described with reference to FIG. 7 may be partially or entirely identical to the components and operations of the components described with reference to FIGS. 1 to 6. The components and operations of the components described with reference to FIG. 7 may be partially or entirely identical to the components and operations of the components described with reference to FIGS. 8 to 17g, which will be described later.

[0128] In the embodiment of FIG. 7, it is assumed that the operation by the prediction module (e.g., operation 520 of FIG. 5) and the operation by the prompt generation module (e.g., operation 530 of FIG. 5)) are performed by an electronic device (e.g., the electronic device (101) of FIG. 1 or the first electronic device (300) of FIG. 3). For example, in the embodiment of FIG. 7, the electronic device (e.g., the electronic device (101) of FIG. 1 or the first electronic device (300) of FIG. 3) may include a prediction module (e.g., the prediction module (380) of FIG. 3) and / or a prompt generation module (e.g., the prompt generation module (390) of FIG. 3), and the server may not include a prediction module (e.g., the prediction module (460) of FIG. 4) and / or a prompt generation module (e.g., the prompt generation module (470) of FIG. 4). When such terminal-based prediction and / or response operations are performed, the delay in communication between the terminal and the server can be reduced compared to server-based methods. However, unlike servers, which continuously collect and store the necessary data for prediction and / or response due to their high computing performance and large storage capacity, electronic devices cannot continuously collect and store the necessary data due to resource constraints. Therefore, to prevent delays in collecting the necessary data, electronic devices must collect the necessary data in advance, either periodically or upon request, before it is used.

[0129] In the embodiment of FIG. 7, for convenience of explanation, the user input data for calling the assistant application is described as an example in which the user input data includes a voice input (701) corresponding to the speech of the first user (U1). However, this is not limited thereto, and the same explanation may be applied even when various types of user input (e.g., text input) are used.

[0130] Referring to FIG. 7, according to one embodiment, the electronic device (300) can receive user input data including voice input (701) through the input interface (310). The electronic device (300) can transmit the user input data to the ASR module (361) through the assistant client (371) within the intelligent module (370).

[0131] According to one embodiment, the electronic device (300) may process user input data including a voice input (701) through the ASR module (361) to obtain user input data including text corresponding to the voice input (701). The electronic device (300) may obtain usage pattern information and / or at least one piece of related information using the prediction module (380). The operation of the electronic device (300) obtaining the usage pattern information and / or at least one piece of related information using the prediction module (380) may include operations 520 and 520 of FIG. 5 , operations 8 and 9 of FIG. The electronic device (300) may generate a prompt using the prompt generation module (390). The operation of the electronic device (300) generating a prompt using the prompt generation module (390) may include operations 530 of FIG. 5 , operations 10A to 10C , and operations 11 of FIG. The electronic device (300) can generate a response to a prompt using a generative AI model (372) (e.g., sLLM, sLMM, LLM, LMM). The operation of the electronic device (300) generating a response to the prompt using the generative AI model (372) may include operation 540 of FIG. 5. The electronic device (300) can process a response including text through a TTS module (364) to obtain a response including a voice corresponding to the text.

[0132] According to one embodiment, the electronic device (300) can transmit the acquired response to the output interface (320) via the assistant client (371). The electronic device (300) can provide a message (702) including the response via the output interface (320). For example, the electronic device (300) can provide the voice included in the response (or message) audibly via a speaker. For example, the electronic device (300) can visually provide the visual information included in the response (or message) via a display.

[0133] FIG. 8 is a flowchart illustrating an operation of an electronic device collecting at least one piece of relevant information based on usage pattern information, according to one embodiment of the present disclosure.

[0134] FIG. 9 is a diagram illustrating an operation of an electronic device collecting at least one piece of relevant information through a prediction module according to one embodiment of the present disclosure.

[0135] The components and operations of the components described with reference to FIGS. 8 and 9 may be partially or entirely identical to the components and operations of the components described with reference to FIGS. 1 to 7. The components and operations of the components described with reference to FIGS. 8 and 9 may be partially or entirely identical to the components and operations of the components described with reference to FIGS. 10 to 17g, which will be described later.

[0136] In the following examples, the operations may be performed sequentially, but are not necessarily sequential. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0137] According to one embodiment, operations 810 to 830 may be understood to be performed in a processor (e.g., processor (340) of FIG. 3 or processor (420) of FIG. 4) of an electronic device (e.g., first electronic device (300) of FIG. 3 or second electronic device (400) of FIG. 4).

[0138] Referring to FIGS. 8 and 9 , according to one embodiment, in operation 810, an electronic device (e.g., the first electronic device (300) of FIG. 3 or the second electronic device (400) of FIG. 4 ) may identify at least one piece of relevant information that needs to be collected based on usage pattern information. For example, if the usage pattern information includes information about a pattern in which a user inquires about the weather through an assistant at a specific time (e.g., 8:00 AM), the electronic device may identify that weather information needs to be collected. For example, if the usage pattern information includes information about a pattern in which a user inquires about a schedule through an assistant at a specific time (e.g., 9:00 AM), the electronic device may identify that schedule information needs to be collected.

[0139] According to one embodiment, in operation 820, the electronic device (300, 400) may transmit (901) a request message to request at least one piece of related information to at least one module (900). In operation 830, the electronic device (300, 400) may receive (902) a response message including at least one piece of related information from at least one module (900) in response to the request message. The at least one module (900) may include, but is not limited to, a first module (e.g., a weather module (911) including a weather application installed in the electronic device (300, 400)) capable of collecting information using an internal service (e.g., an application installed in the electronic device (300, 400)), and / or a second module (e.g., an external search module (913) including a web application) capable of collecting information by searching the outside (920) (e.g., the web). According to one embodiment, the first module can use information stored inside the electronic device (300, 400) as well as information outside the electronic device (300, 400), but the second module cannot use information stored inside the electronic device (300, 400).

[0140] According to one embodiment, when the usage pattern information is associated with first information (e.g., weather information and / or schedule information) that can be collected using an internal service, the electronic device (300, 400) can transmit a request message requesting the first information to a first module (e.g., weather module (911) and / or schedule module (912)) and receive a response message including the first information from the first module, respectively. When the usage pattern information is associated with second information (e.g., stock information) that can be collected only through an external search, the electronic device (300, 400) can transmit a request message requesting the second information to an external search module (913) and receive a response message including the second information from the external search module (913).

[0141] According to one embodiment, information collectible using internal services may be synchronized and stored on each electronic device (300, 400). For example, schedule information of a schedule application installed on the electronic device (300) may be periodically synchronized and the same information may be stored on each electronic device (300, 400). In this case, each electronic device (300, 400) may not transmit a request to obtain the corresponding information to other electronic devices. Meanwhile, some of the information collectible using internal services may not be synchronized and stored on each electronic device (300, 400) but may be stored only on one electronic device. For example, health information of a health application installed on the electronic device (300) may not be synchronized with other electronic devices (400) for privacy protection and may be stored only on the electronic device (300). In this case, the other electronic device (400) may request the corresponding information from the electronic device (300) when necessary.

[0142] According to one embodiment, the above-described operations 810 to 830 may be performed by the prediction module (380, 460) of the electronic device (300, 400). The prediction module (380, 460) may transmit at least one piece of related information acquired through operations 810 to 830 together with usage pattern information to the prompt generation module (390, 470). The prompt generation module (390, 470) may generate a prompt based on the usage pattern information and the at least one piece of related information and transmit the prompt to the intelligent module (370, 450) (e.g., a generative AI module within the intelligent module (370, 450)).

[0143] FIG. 10A is a flowchart illustrating an operation of an electronic device generating a prompt according to one embodiment of the present disclosure.

[0144] FIG. 10b is a flowchart illustrating a procedure including a prompt generation operation by an electronic device according to one embodiment of the present disclosure.

[0145] FIG. 10c is a flowchart illustrating a procedure including a prompt generation operation by a plurality of electronic devices according to one embodiment of the present disclosure.

[0146] FIG. 11 is a diagram illustrating an operation of an electronic device generating a prompt through a prompt generation module according to one embodiment of the present disclosure.

[0147] The components and operations of the components described with reference to FIGS. 10a to 10c and 11 may be partially or entirely identical to the components and operations of the components described with reference to FIGS. 1 to 9. The components and operations of the components described with reference to FIGS. 10a to 10c and 11 may be partially or entirely identical to the components and operations of the components described with reference to FIGS. 12 to 17g, which will be described later.

[0148] In the following examples, the operations may be performed sequentially, but are not necessarily sequential. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0149] According to one embodiment, operations 1010a to 1030a may be understood to be performed in a processor (e.g., processor (340) of FIG. 3 or processor (420) of FIG. 4) of an electronic device (e.g., first electronic device (300) of FIG. 3 or second electronic device (400) of FIG. 4).

[0150] Referring to FIGS. 10A to 10C and 11, according to one embodiment, in operation 1010a, an electronic device (e.g., electronic device (300) of FIG. 3 or electronic device (400) of FIG. 4) may obtain at least one prompt information associated with the content of a response. For example, the electronic device (300, 400) may obtain at least one prompt information based on request information (1110), first analysis information (1121), and / or at least one prompt template (1130). For example, as illustrated in FIG. 11, request information (1110) may be received from an assistant application (371) and may correspond to a user request portion (e.g., user query or user request) included in user input data. For example, as illustrated in FIG. 11, the first analysis information (1121) is received from the prediction module (380, 460) and may include usage pattern information and / or at least one piece of related information.

[0151] According to one embodiment, the at least one prompt information may include first prompt information including at least one of a prediction query or a prediction request obtained based on first analysis information (1121) (e.g., usage pattern information) and / or second prompt information including at least one of a user request or a user query obtained based on request information (1110) (e.g., user input data).

[0152] According to one embodiment, in operation 1020a, the electronic device (300, 400) may obtain at least one piece of expression method information associated with the expression method of the response. For example, the electronic device (300, 400) may obtain at least one piece of expression method information based on at least one piece of prompt information and / or second analysis information (1122). For example, as illustrated in FIG. 11, the second analysis information (1122) may be received from the prediction module (380, 460) and may include at least one piece of utterance analysis information. At least one piece of speech analysis information is obtained by analyzing the content of the user's speech through the assistant application using the prediction module (380, 460), and may include, but is not limited to, for example, the method of calling the assistant application (e.g., voice call, button input), the time of calling the assistant application (e.g., morning, evening), the location (or place) of calling the assistant application (e.g., home, work), the acoustic characteristics of the user's speech through the assistant application (e.g., calling in a loud, soft, high, low voice), and / or the linguistic characteristics of the user's speech through the assistant application (e.g., speech pattern, intonation, sentence length, words used, sentences). The acoustic characteristics may include, but are not limited to, characteristics associated with the physical properties of a voice (e.g., voice) (e.g., waveform, amplitude, frequency, timing, spectrum). Linguistic features can include, for example, features related to the structure and meaning of language (e.g., phonetics, vocabulary, grammar, syntax, preferred word and sentence structure, speech style (or tone), sentence length). By considering various factors in determining the response expression, natural responses tailored to the user's characteristics, environment, situation, and preferences can be provided.

[0153] According to one embodiment, the at least one expression mode information may include first expression mode information for the expression mode of a first response associated with the first prompt information and / or second expression mode information for the expression mode of a second response associated with the second prompt information.

[0154] In one embodiment, the second presentation method information may be different from the first presentation method information. For example, the first presentation method information may include information for indirectly expressing a first response associated with the first prompt information, and the second presentation method information may include information for directly expressing a second response associated with the second prompt information. In this case, even if the first and second responses are responses providing the same content (e.g., a response providing today's weather information), the first response based on prediction may be expressed indirectly (e.g., "It's a day that makes me feel good because the weather is nice."), and the second response, which is a response to a direct query or request based on user input, may be expressed directly (e.g., "It's a sunny day with a temperature of 30 degrees."). This method allows for a natural response to be provided to the user. For example, although the first response based on prediction is based on past usage patterns, it may be a response that the user does not actually want at the present time, and expressing it directly like the second response may cause the user to feel uncomfortable. However, if you express it indirectly, you can provide a response to the user without making it seem awkward, even if it is not the response the user actually wants at the moment.

[0155] According to one embodiment, in operation 1030a, the electronic device (300, 400) may generate a prompt (1140) based on at least one prompt information and / or at least one expression method information. The electronic device (300, 400) may transmit the acquired prompt (1140) to the generative AI model (372, 452).

[0156] According to one embodiment, the electronic device (300, 400) may generate prompts in stages. For example, the electronic device (300, 400) may generate at least one first prompt (e.g., a first prompt) including first prompt information and second prompt information acquired through operation 1010a, and after the at least one first prompt is generated, the electronic device (300, 400) may generate a second prompt (e.g., a second prompt) further including first expression method information and second expression method information acquired through operation 1020a. In this case, the electronic device (300, 400) may transmit the second prompt, which reflects at least one expression method information in at least one first prompt, to the generative AI model (372, 452).

[0157] Table 1 below shows examples of information and prompt templates used to generate prompts.

[0158] Type Content Request Information (1110) <request>Article information search (e.g., "Hi Bixby, search for article information") 1st analysis information (1121)<pattern_type> Search for today's Seoul weather information. Second analysis information (1122) Speech tone, sentence length, and time of speech (e.g., "Hi Bixby, tell me the weather (today)" or "Hi Bixby, weather"). Prompt template 1<pattern_type> Please do this prompt template 2 <request>Please summarize it

[0159] According to one embodiment, the electronic device (300, 400) can generate a prompt using at least one prompt template (1130). The prompt template (1130) may be set for each of the request information (1110) (e.g., spoken content) and the first analysis information (1121) (e.g., predicted / analyzed content), as exemplified in Table 1, but is not limited thereto. For example, one prompt template (1130) may be set for both the request information (1110) (e.g., spoken content) and the first analysis information (1121) (e.g., predicted / analyzed content).

[0160] According to one embodiment, when the information and prompt template (1130) used to generate a prompt are as exemplified in Table 1, the electronic device (300, 400) can generate a primary prompt such as “Search for today’s Seoul weather information” using the first analysis information (1121) and prompt template 1, and can generate a primary prompt such as “Summarize today’s article” using the request information (1110) and prompt template 2.

[0161] According to one embodiment, the electronic device (300, 400) may generate a secondary prompt using the primary prompt(s) and the secondary analysis information (1122). The secondary analysis information (1122) may be set to different values, for example, depending on the length of the spoken sentence or the speech tone. For example, if the content of the utterance that calls the assistant application ends in a long sentence (e.g., a long sentence), such as "Hi Bixby, tell me the weather (today)", the secondary analysis information (1122) may include information that causes the corresponding response to be generated with a long length. In this case, the secondary prompt may be as follows. The secondary prompt may be generated as follows: "Search for today's weather information in Seoul. Express the above content indirectly at length. Express today's article in a summarized manner. Answer by reflecting the overall speech expression." For example, if the content of the utterance that calls the assistant application ends with a short word such as "Hi Bixby, weather" (e.g., a short sentence), the second analysis information (1122) may include information that causes the corresponding response to be generated with a shorter length. In this case, the secondary prompt may be as follows. A secondary prompt such as "Search for today's weather information in Seoul. Express the above content indirectly and briefly. Summarize today's article. Answer by reflecting the overall expression of the utterance." may be generated.

[0162] According to one embodiment, electronic devices (300, 400) can cooperate with each other to generate prompts in a stepwise manner. For example, a second electronic device (400) (e.g., a server) can generate at least one first prompt (e.g., a first prompt) including first prompt information and second prompt information obtained through operation 1010a, and the first electronic device (300) can generate a second prompt (e.g., a second prompt) further including first expression method information and second expression method information obtained through operation 1020a after generating at least one first prompt. According to one embodiment, the second electronic device (400) may generate at least one first prompt and / or second analysis information (1122) and transmit them to the first electronic device (300), and the first electronic device (300) may generate at least one expression method information based on the first prompt and / or second analysis information (1122), and may add at least one expression method information to at least one first prompt to generate a second prompt. In this case, the first electronic device (300) may transmit the second prompt, which reflects at least one expression method information to at least one first prompt, to the generative AI model (372). Since the features associated with the expression method of the response are relatively small in scale compared to the features associated with the content of the response, analysis is also possible through the generative AI model (e.g., sLLM) of the first electronic device (300), and generation of a response to the corresponding prompt is also possible. Therefore, even if the generation of the second prompt and the generation of the response based on the second prompt are performed on the electronic device rather than on the server, processing performance is not an issue.

[0163] In one embodiment, the electronic device (300, 400) may generate a single prompt. For example, the electronic device (300, 400) may generate a single prompt including the first prompt information, the second prompt information, the first expression method information, and / or the second expression method information obtained through operations 1010a to 1020a. In this case, the electronic device (300, 400) may transmit the generated prompt to the generative AI model (372, 452).

[0164] In one embodiment, the electronic device (300, 400) may generate a prompt using a learned artificial intelligence model instead of using a prompt template. For example, the electronic device (300, 400) may input input data including request information (1110), first analysis information (1121), and / or second analysis information (1122) into the learned artificial intelligence model and obtain a prompt (1140) output from the artificial intelligence model. In this case, a prompt (1140) may also be generated for content not configured as a prompt template.

[0165] FIG. 10b illustrates a procedure including a prompt generation operation by a second electronic device (hereinafter, “server”) (400). In the embodiment of FIG. 10b, an example is provided in which both a primary prompt and a secondary prompt are generated by the server (400). In this case, the first electronic device (hereinafter, “electronic device”) (300) can provide a response generated and transmitted from the server (400) to the user without performing separate prompt generation and response generation operations.

[0166] Referring to FIG. 10b, in operation 1010b, the electronic device (300) may activate an assistant application. Operation 1010b may include operation 510 of FIG. 5. In operation 1020b, the electronic device (300) may transmit first data to the server (400). The first data may be, for example, user input data including the voice input (601) of FIG. 6. In operations 1030b and 1040b, the server (400) may sequentially generate a first prompt and a second prompt based on the first data. Operations 1030b and 1040b may include, for example, operations 1010a to 1030a of FIG. 10a. In operation 1050b, the server (400) may transmit second data to the electronic device (300). The second data may include, for example, a response to a second prompt. The response to the second prompt may be generated using a generative AI model (452) of the server (400). In operation 1060b, the electronic device (300) may provide a response.

[0167] FIG. 10C illustrates a procedure including a prompt generation operation by an electronic device (300) and a server (400). In the embodiment of FIG. 10C, an example is provided in which a primary prompt is generated by the server (400) and a secondary prompt is generated by the electronic device (300). In this case, rather than providing a response generated and transmitted from the server (400) to the user as is, the electronic device (300) may provide a response modified by the electronic device (300) (e.g., modification of the expression method, such as the tone and length of the response) to the user.

[0168] Referring to FIG. 10c, in operation 1010c, the electronic device (300) may activate an assistant application. Operation 1010c may include operation 510 of FIG. 5. In operation 1020c, the electronic device (300) may transmit first data to the server (400). The first data may be, for example, user input data including the voice input (601) of FIG. 6. In operation 1030c, the server (400) may generate a first prompt based on the first data. Operation 1030c may include, for example, at least some of operations 1010a to 1030a of FIG. 10a. In operation 1040c, the server (400) may transmit second data to the electronic device (300). The second data may include, for example, a first prompt, a response to the first prompt, and / or second analysis information (e.g., the second analysis information (1122) of FIG. 11 ). The response to the first prompt may be generated using the generative AI model (452) of the server (400). At operation 1050c, the electronic device (300) may generate the second prompt based on the second data. In one embodiment, the electronic device (300) may generate the response to the second prompt using the generative AI model (372). At operation 1060c, the electronic device (300) may provide the response.

[0169] FIG. 12 is a diagram illustrating a response to a user input according to one embodiment of the present disclosure.

[0170] The components and operations of the components described with reference to FIG. 12 may be partially or entirely identical to the components and operations of the components described with reference to FIGS. 1 to 11. The components and operations of the components described with reference to FIG. 12 may be partially or entirely identical to the components and operations of the components described with reference to FIGS. 13a to 17g, which will be described later.

[0171] Part (a) of FIG. 12 illustrates a response to a user input (e.g., voice input) when using a conventional method that does not predict a usage pattern of an assistant application, and part (b) of FIG. 12 illustrates a response to a user input (e.g., voice input) when using the proposed method of the present disclosure that predicts a usage pattern of an assistant application.

[0172] Referring to part (a) of FIG. 12, a conventional response to the user input “Hi Bixby” may include a single response such as “How may I help you?”

[0173] Referring to part (b) of Fig. 12, the proposed method's response to the same user input "Hi Bixby" can be one of various responses such as "It's a sunny day, how can I help you?", "It's a sunny day, how can I help you?", "It's a clear blue day, how can I help you?", "The sky is clear without a single cloud, how can I help you?", "It's a day with good weather and a good mood, how can I help you?". In this way, when using the proposed method, even if the user input is the same, various responses can be provided depending on the configuration of various prompts according to the prediction of the usage pattern and / or speech pattern.

[0174] FIGS. 13A to 13C are diagrams illustrating responses to user input based on analysis of usage patterns of an assistant application according to one embodiment of the present disclosure.

[0175] The components and operations of the components described with reference to FIGS. 13a to 13c may be partially or entirely identical to the components and operations of the components described with reference to FIGS. 1 to 12. The components and operations of the components described with reference to FIGS. 13a to 13c may be partially or entirely identical to the components and operations of the components described with reference to FIGS. 14 to 17g, which will be described later.

[0176] Figure 13a illustrates usage patterns (e.g., query patterns) of a user's assistant application.

[0177] Referring to FIG. 13a, the response to the first user input (query 1), “Hi Bixby, tell me today’s weather,” is a response to a direct weather query, and thus includes content such as “It’s expected to rain all day today, so be careful of the indoor humidity. The lowest temperature is 22 degrees, and the highest temperature is 27 degrees.”, which directly provides actual weather information; the response to the second user input (query 2), “Read me the news.” includes “Shall I read you the most recent article?”; the response to the third user input (query 3), “Read me yesterday’s article.” includes “Yesterday’s last article was ~.”; the response to the fourth user input (query 4), “Tell me about today’s stocks.” includes “Which company’s stock price should I tell you about?”; and the response to the fifth user input (query 5), “How much is OO stock?” can include “OO Electronics is priced at 80,000 won as of 9 o’clock.” In this case, the usage pattern of the assistant application may have a pattern that includes multiple consecutive queries (e.g., weather (query 1) → news (query 2) → article (query 3) → stock (query 4) → OO electronics stock (query 5)).

[0178] Figure 13b illustrates a response provided in a state in which the usage pattern of Figure 13a has been learned (e.g., learned through three repetitions of the same usage pattern).

[0179] Referring to FIG. 13b, the response to the first user input (query 1), “Hi Bixby, tell me the weather today,” includes “It’s expected to rain all day today, so be mindful of the indoor humidity. The lowest temperature is 22 degrees, and the highest temperature is 27 degrees.” The response to the second user input (query 2), “Read me the news.”, unlike FIG. 13a, omits the response “Would you like me to read you the most recent article?” and includes a response such as “Yesterday’s last article was ~.” The response to the third user input (query 4), “Tell me about today’s stocks.”, unlike FIG. 13a, omits the response “Which company’s stock price would you like me to tell you?” and includes “OO Electronics is 80,000 won as of 9 o’clock.” In this way, when the usage pattern is learned at the first level, the user can immediately receive the answers he ultimately wants through the intermediate queries (queries 3 and 5) and the corresponding queries (queries 1, 2 and 4) without any response to them through prediction based on the usage pattern.

[0180] Figure 13c illustrates a response provided when the usage pattern of Figure 13a has been learned beyond a specified standard (e.g., learned through 10 repetitions of the same usage pattern).

[0181] Referring to FIG. 13c, the response to the first user input (query 1), "Hi Bixby, tell me the weather today," may include "It's expected to rain all day today, so be mindful of the indoor humidity. The lowest temperature is 22 degrees and the highest temperature is 27 degrees. The last article yesterday was ~. The last article yesterday was ~." Or, the response to the first user input (query 1), "Hi Bixby, tell me the weather today," may include "It's a cloudy day starting at 80,000 won for OO Electronics as of 9 o'clock. It's expected to rain all day today, so be mindful of the indoor humidity. The lowest temperature is 22 degrees and the highest temperature is 27 degrees. The last article yesterday was ~. Would you like me to read more?" In this way, if the usage pattern is learned to a certain level or higher (e.g., a second level or higher than the first level), the entire response according to the user's desired usage pattern can be provided at once without separate additional queries (queries 2 to 5) through prediction according to the usage pattern based on the first query (query 1). In other words, the entire response the user wants can be provided at once with only the first query or call to the assistant application. In one embodiment, each response in the entire response can be provided sequentially according to the usage pattern. This allows the user to receive the desired response without the cumbersome process of repeating the query. Additionally, the electronic device can provide a link to check additional information about the response (e.g., the entire article content, stock information), along with the response, and the user can check the additional information by accessing the link.

[0182] FIG. 14 is a diagram illustrating a response to a user input according to a usage pattern of an assistant application according to one embodiment of the present disclosure.

[0183] The components and operations of the components described with reference to FIG. 14 may be partially or entirely identical to the components and operations of the components described with reference to FIGS. 1 to 13c. The components and operations of the components described with reference to FIG. 14 may be partially or entirely identical to the components and operations of the components described with reference to FIGS. 15 to 17f, which will be described later.

[0184] Part (a) of Fig. 14 illustrates a response (e.g., a welcome message) to a user input including a wake word portion (e.g., “Hi Bixby”) and a request portion (e.g., “Today’s weather”) before learning the usage pattern.

[0185] Referring to part (a) of FIG. 14, a response to the user input “Hi Bixby, today’s weather” may include content such as “It is a sunny day with a low of 15 degrees and a high of 23 degrees,” which directly provides actual weather information, since the user input includes a direct weather query.

[0186] Part (b) of Fig. 14 illustrates a response (e.g., a welcome message) to a user input that includes only a wake word part (e.g., “Hi Bixby”) without a request part (e.g., “Today’s weather”), in a state where the usage pattern of part (a) of Fig. 14 has been learned.

[0187] Referring to part (b) of FIG. 14, the response to the user input "Hi Bixby" may include content such as "Do you need any help on this sunny day?" or "Fighting like today's sunny weather!" through learning of usage patterns. In this way, responses predicted based on usage patterns can be indirectly provided to the user without a separate, explicit query or request.

[0188] Part (c) of Fig. 14 illustrates a response (e.g., a welcome message) to a user input including a request portion (e.g., today's news) and a wake word portion (e.g., Hi Bixby), which is different from the request portion (e.g., today's weather) of part (a) of Fig. 14, which is a request portion according to the learned usage pattern, in a state where the usage pattern of part (a) of Fig. 14 has been learned.

[0189] Referring to section (c) of FIG. 14, the response to the user input "Hi Bixby, today's news" may include a predicted response based on the usage pattern, such as "Like the sunny weather today, the first article of today is ~." In this way, along with the directly requested response, a predicted response based on the usage pattern can be indirectly provided to the user.

[0190] FIG. 15 is a diagram illustrating a method for an electronic device to provide a response using a gesture input, according to one embodiment of the present disclosure.

[0191] The components and operations of the components described with reference to FIG. 15 may be partially or entirely identical to the components and operations of the components described with reference to FIGS. 1 to 14. The components and operations of the components described with reference to FIG. 15 may be partially or entirely identical to the components and operations of the components described with reference to FIGS. 16 to 17g, which will be described later.

[0192] In the embodiment of FIG. 15, for convenience of explanation, an example is provided in which a user wearing an electronic device (1500) of the type of head mounted display (HMD) device (e.g., augmented reality (AR) glasses) (e.g., the electronic device (101) of FIG. 1 or the first electronic device (300) of FIG. 3) calls an assistant application and includes a gesture input (e.g., the user's gaze input or hand gesture input) as part of the user input (1510). However, the same explanation may also be applied to other types of electronic devices (e.g., a smart watch) and / or other types of input (e.g., text input).

[0193] Referring to FIG. 15, a user input (1510) may include a wake word portion (e.g., “Hi Bixby”), a portion corresponding to an eye gaze, and / or a user request portion such as “Search for me.” According to one embodiment, the electronic device (1500) may activate an assistant application based on the wake word portion of the user input (1510), identify an object (1511) (e.g., a table) at which the user’s gaze is located, and generate a prompt requesting a search for the identified object. The electronic device (1500) may provide (e.g., display) a response (1520) to the user input (1510) based on the generated prompt. The response (1520) may include a sentence such as “This is a ** table from Brand A. It’s similar to the table I’ve been interested in recently!” In this way, the electronic device (1500) may provide a response desired by the user by using a separate input (e.g., a gesture input) in addition to a voice input.

[0194] FIG. 16 is a diagram illustrating a method for an electronic device to provide a response including visual information, according to one embodiment of the present disclosure.

[0195] FIGS. 17A to 17G are diagrams illustrating responses including visual information according to one embodiment of the present disclosure.

[0196] The components and operations of the components described with reference to FIGS. 16 and 17a to 17g may be partially or entirely identical to the components and operations of the components described with reference to FIGS. 1 to 15.

[0197] In the embodiments of FIGS. 16 and 17A to 17G, an electronic device (e.g., the electronic device (101) of FIG. 1 or the first electronic device (300) of FIG. 3) may provide a response (or message) that includes visual information along with audio information and / or a screen display. Through such visual information, a more detailed and intuitive response may be provided to the user.

[0198] Referring to part (a) of FIG. 16, the electronic device may display a screen including an interface (1610) (e.g., a card interface) including related content (1611) and / or an image (1612) along with a voice output and / or a screen display in response. According to one embodiment, as illustrated in part (a) of FIG. 17a, a response to the input “Hi Bixby, today’s weather” may include a card interface (1710a) including related content (1711a) and a weather image (e.g., a sun image) (1712a) along with a voice output and / or a screen display such as “The lowest temperature is 15 degrees and the highest temperature is 22 degrees.”

[0199] According to one embodiment, as illustrated in part (a) of FIG. 17b, in a state where the usage pattern of FIG. 17a is learned, a response to the input of “Hi Bixby, today’s news” may include a card interface (1710b) including an article title (e.g., AI’s opportunity to counterattack) (1711b), summary information about the article, and / or an article image (e.g., a representative face of a company associated with the article) (1712b), along with a voice output and / or screen display such as “Like today’s sunny weather, the first article of today that makes me feel good is ~” that includes an indirect expression for weather information and a direct expression for article information.

[0200] Referring to part (b) of FIG. 16, the electronic device may display a screen including an interface (1620) (e.g., a card interface) including related content (1611), an image (1612), and an entry point (1613) of a related domain (e.g., an entry point of a weather application), along with voice output and / or screen display. When the entry point (1613) of the related domain is selected, the electronic device may switch the screen to the screen of the corresponding domain. According to one embodiment, the related domain (e.g., a related application) may correspond to a query predicted based on a usage pattern and may be set by the user. For example, the electronic device may set at least one application (e.g., a weather application, an article application, a video application) as an application (e.g., a default application) that provides additional information corresponding to the predicted query based on user settings and taking into account the operating speed of the software. For example, the electronic device may set at least one application (e.g., a weather application, an article application, a video application) preferred by the user based on user settings as an application that provides additional information corresponding to the predicted query. An embodiment of setting at least one application to be used, for example, for a welcome message, based on user settings is described below with reference to FIGS. 18a to 18b.

[0201] According to one embodiment, as illustrated in part (b) of FIG. 17a, a response to the input “Hi Bixby, today’s weather” may include a card interface (1720a) including weather information (e.g., a sunny day), a weather image (e.g., a sun image), and a weather application entry point (1713a) (e.g., a “Go to Weather App” button), along with a voice output and / or screen display such as “The lowest temperature is 15 degrees and the highest temperature is 22 degrees.” The electronic device may switch to the weather app when the “Go to Weather App” button is selected.

[0202] According to one embodiment, as illustrated in part (b) of FIG. 17b, in a state where the usage pattern of FIG. 17a is learned, a response to the input of “Hi Bixby, today’s news” may include a card interface (1720b) ​​including an article title (e.g., seeking a counterattack opportunity with AI), an article image (e.g., a representative face of a company associated with the article), and a news application entry point (1713b) (e.g., a “Go to News App” button), along with a voice output and / or screen display such as “Like today’s sunny weather, the first article of the day that feels good is ~” that includes an indirect expression for weather information and a direct expression for article information. The electronic device may switch to the news app when the “Go to News App” button is selected.

[0203] Referring to part (c) of FIG. 16, the electronic device may display a screen including an interface (1630) (e.g., a card interface) including related content (1611), an image (1612), an entry point (1613) of a related domain (e.g., an entry point of a weather application), and user recommended content (1614) (e.g., a recommended video related to weather), along with voice output and / or screen display.

[0204] According to one embodiment, as illustrated in part (c) of FIG. 17a, a response to the input of “Hi Bixby, today’s weather” may include a card interface (1730a) including weather information (e.g., a sunny day), a weather image (e.g., a sun image), a weather application entry point (1713a) (e.g., a “Go to Weather App” button), and weather-related recommendations (1714a) (e.g., a weather-related recommended video), along with a voice output and / or screen display such as “The lowest temperature is 15 degrees and the highest temperature is 22 degrees.” The electronic device may switch to the video app when the execution button of the “weather-related recommended video” is selected.

[0205] According to one embodiment, as illustrated in part (c) of FIG. 17b, in a state where the usage pattern of FIG. 17a is learned, a response to the input of “Hi Bixby, today’s news” may include a voice output and / or screen display such as “Like the sunny weather today, the first article today that feels good is ~” that includes an indirect expression for weather information and a direct expression for article information, and a card interface (1730b) that includes an article title (e.g., looking for a counterattack opportunity with AI), a summary of the article content, an article image (e.g., a representative face of a company related to the article), a news application entry point (1713b) (e.g., a “Go to News App” button), and news-related recommendation content (1714b) (e.g., stock price information of the company and a “Go to Stock App” button). When the “Go to Stock App” button is selected, the electronic device may switch to the stock app. According to one embodiment, the electronic device may select an article related to weather information based on usage pattern information and include information about the selected article (e.g., article title, article summary, article image, and information about the entry point to a news application for the article) in the card interface (1730b). For example, articles related to weather information may include cheerful articles when the weather is sunny, and may include depressing articles or uplifting articles when the weather is cloudy or rainy.

[0206] Figure 17c illustrates an example of sequentially providing responses including a card interface according to a learned usage pattern (e.g., the usage pattern of Figure 13a). When a response (or message) based on a learned usage pattern is output, the electronic device can configure a visual interface (e.g., a card interface) centered on the first query, and then sequentially provide the visual interface according to the subsequent queries.

[0207] According to one embodiment, a response to the input “Hi Bixby, tell me the weather today” may include a card interface (1710c) including weather information (e.g., a sunny day), a weather image (e.g., a sun image), and a weather application entry point (e.g., a “Go to the weather app” button), along with a voice output and / or screen display such as “The lowest temperature is 15 degrees and the highest temperature is 22 degrees,” as illustrated in part (a) of FIG. 17c. Thereafter, a response to the input “Tell me the news” may include a card interface (1720c) including an article title (e.g., looking for a counterattack opportunity with AI), a summary of the article content, an article image (e.g., a representative face of the company), and a news application entry point (e.g., a “Go to the news app” button), along with a voice output and / or screen display such as “The first article today is about ~,” as illustrated in part (b) of FIG. Thereafter, the response to the input of "Stock Information" may include a card interface (1730c) including stock price information (e.g., OO Electronics, stock price: 80,200), stock price image (e.g., stock chart), and stock application entry point (e.g., "Go to Stock App" button), along with voice output and / or screen display such as "OO Electronics is at 80,000 won as of 9 o'clock. Should I show you other stocks?", as illustrated in part (c) of FIG. 17c.

[0208] FIG. 17d illustrates an example in which, when a usage pattern (e.g., the usage pattern of FIG. 13a) has been learned to a certain level or higher, responses including a card interface are sequentially provided according to the usage pattern without any additional queries after the first query. When a response (or message) with a learned usage pattern is output, the electronic device can sequentially provide a visual interface (e.g., a card interface) according to the usage pattern on each screen based on the first query.

[0209] According to one embodiment, a response to the input "Hi Bixby, tell me the weather today" may be provided, as illustrated in FIG. 17d, with a voice output and / or screen display such as "Fighting for a sunny day today! Today's first article is about ~", and a card interface (1710d) corresponding to the first query content, a card interface (1720d) corresponding to the predicted first query content, and a card interface (1730d) corresponding to the predicted second query content, without a separate additional query. In the embodiment of FIG. 17d, since only one card interface is displayed on one screen, the user can swipe the screen using a swipe input (1701d) to check the remaining card interfaces that are not currently displayed on the screen.

[0210] FIG. 17e illustrates an example in which, when a usage pattern (e.g., the usage pattern of FIG. 13a) has been learned to a certain level or higher, responses including card interfaces are sequentially provided on a single screen according to the usage pattern without any additional queries after the first query. When a response (or message) with a learned usage pattern is output, the electronic device can sequentially provide visual interfaces (e.g., card interfaces) according to the usage pattern on a single screen based on the first query.

[0211] According to one embodiment, a response to the input "Hi Bixby, tell me the weather today" may be provided, as illustrated in FIG. 17e, with a voice output and / or screen display such as "Fighting for a beautiful day today! Today's first article is about ~.", for example, a card interface (1710e) corresponding to the first query content, a card interface (1720e) corresponding to the predicted first query content, and a card interface (1730e) corresponding to the predicted second query content, without a separate additional query. In the embodiment of FIG. 17e, all or part of the card interfaces (1710e, 1720e, 1730e) may be displayed on one screen. The user may check card interfaces that are not currently displayed on the screen through a scroll input (1701e).

[0212] FIG. 17f illustrates an example in which, in a multi-foldable electronic device (e.g., 3-stage), a usage pattern (e.g., the usage pattern of FIG. 13a) is learned to a certain level or higher, a response including a card interface is sequentially provided according to the usage pattern without any additional queries after the first query.

[0213] According to one embodiment, in the folded state, a response to the input "Hi Bixby, tell me the weather today" may display a first screen (1701f) including a first card interface corresponding to the first query content, along with a voice output and / or screen display such as "Fighting, sunny weather today! Today's first article is about ~." In the folded state, one card interface may be displayed on one screen, as in the embodiment of FIG. 17d. According to one embodiment, in the folded state, the electronic device may provide the user with information notifying that there is additional information. In this case, the user may switch from the folded state to the unfolded state to check the additional information.

[0214] According to one embodiment, when switching from a folding state to an unfolding state, a first screen (1701f) including a first card interface corresponding to the first query content, a second screen (1702f) including a second card interface corresponding to the predicted first query content, and a third screen (1703f) including a third card interface corresponding to the predicted second query content may be provided without a separate additional query, along with a voice output and / or screen display such as "Fighting, sunny weather today! Today's first article is about ~." In the unfolding state, unlike the embodiment of FIG. 17d, a plurality of card interfaces may all be displayed through a plurality of screens.

[0215] FIG. 17g illustrates an example in which, in a rollable electronic device, a response including a card interface is sequentially provided according to a usage pattern without any additional queries after the first query, when a usage pattern (e.g., the usage pattern of FIG. 13a) has been learned to a certain level or higher.

[0216] According to one embodiment, in the first state, a response to the input "Hi Bixby, tell me the weather today" may display a first screen (1701g) including a first card interface corresponding to the first query content, along with a voice output and / or screen display such as "Fighting, sunny weather today! Today's first article is about ~." According to one embodiment, in the first state, the electronic device may provide the user with information notifying that there is additional information. In this case, the user may expand the bottom or top of the display in the first state to check the additional information.

[0217] According to one embodiment, when transitioning from the first state to the second state in which the bottom or top is expanded, a second screen (1702g) including a first card interface corresponding to the first query content, a second card interface corresponding to the predicted first query content, and a third card interface corresponding to the predicted second query content can be provided without a separate additional query, along with a voice output and / or screen display such as "Fighting, sunny weather today! Today's first article is about ~."

[0218] FIGS. 18A to 18C are diagrams illustrating screens for setting a welcome message according to one embodiment of the present disclosure.

[0219] Referring to FIG. 18a, an electronic device (e.g., the electronic device (101) of FIG. 1 or the first electronic device (300) of FIG. 3) can display a first screen (1800a) for setting a welcome message (1811).

[0220] According to one embodiment, the first screen (1800a) may provide a settings interface (1810) including a first setting (1812) for selecting an app to use in the welcome message and / or a second setting (1813) for selecting a user screen for the welcome message.

[0221] In one embodiment, the first screen (1800a) may include a first screen portion (1812a) on which at least one app selected by the first setting (1812) is displayed. Since the app selected by the first setting (1812) does not yet exist on the first screen (1800a), the first screen portion (1812a) may not display an image corresponding to the app (e.g., an app icon).

[0222] According to one embodiment, the first screen (1800a) may include a second screen portion (1813a) displaying user screens for welcome messages configurable by the second setting (1812). For example, the second screen portion (1813a) may include a first selection item (1814a) for a first user screen (e.g., a screen including the interface (1610) of FIG. 16), a second selection item (1814b) for a second user screen (e.g., a screen including the interface (1620) of FIG. 16), and / or a third selection item (1814c) for a third user screen (e.g., a screen including the interface (1630) of FIG. 16).

[0223] Referring to FIG. 18b, the electronic device may display a second screen (1800b) for setting an app used for a welcome message. The second screen (1800b) may be displayed in response to receiving a user input (e.g., a touch input) for setting a first setting (1812) on the first screen (1800a).

[0224] According to one embodiment, the second screen (1800b) may provide an app settings interface (1820) including a first setting (1821) for adding (or selecting) an app to be used in the welcome message and / or a second setting (1822) for setting the order of the added apps.

[0225] In one embodiment, the second screen (1800b) may display at least one app selected or added by the first setting (1821). For example, as illustrated in FIG. 18b, the Contacts app, the Calendar app, and the Phone app may be displayed as being used for the welcome message.

[0226] According to one embodiment, the second screen (1800b) may display the order in which apps selected on the user screen are output (or displayed) as set by the second setting (1822). For example, as illustrated in FIG. 18b, the contact app, calendar app, and phone app may be displayed on the user screen in that order, and this display order may be changed via the second setting (1822).

[0227] Referring to FIG. 18c, the electronic device may display a third screen (1800c) showing settings for a welcome message. The third screen (1800c) may be displayed in response to receiving a user input (e.g., a touch input) on the second screen (1800b) selecting to apply the settings to the app settings interface (1820).

[0228] In one embodiment, the third screen (1800c) may include one or more of the configurations displayed on the first screen (1800a). For example, the third screen (1800c) may provide a settings interface (1810) including a first setting (1812) for selecting an app to use in the welcome message and / or a second setting (1813) for selecting a user screen for the welcome message.

[0229] According to one embodiment, the first screen portion (1812c) of the third screen (1800c) may display an image corresponding to at least one app selected by the first setting (1812). For example, the first screen portion (1812c) may display images corresponding to a contacts app, a phone app, and a calendar app, respectively.

[0230] In one embodiment, the second screen portion (1813c) of the third screen (1800c) may display a user screen selected from among the user screens. For example, the second screen portion (1813c) may indicate that the first user screen corresponding to the first selection item (1814a) is selected.

[0231] According to one embodiment of the present disclosure, an electronic device (e.g., the first electronic device (300) of FIG. 3 or the second electronic device (400) of FIG. 4) can analyze a user's usage pattern of an assistant application (e.g., a pattern of queries repeated at a specific time), identify expected queries or requests based on the analysis results, and generate a response to the identified queries or requests. Through this, the electronic device can provide the user with a response to an expected query or request based on the usage pattern, even if there is no direct query or request.

[0232] According to one embodiment, an electronic device can analyze user speech patterns (e.g., tone of voice, sentence length) via an assistant application, determine a response format based on the analysis results, and generate a response based on the determined format. This allows the electronic device to provide a natural-sounding response to the user by expressing a response in a format determined based on the speech pattern, even without a direct request for a specific format.

[0233] According to one embodiment of the present disclosure, an electronic device (electronic device 101 of FIG. 1, a first electronic device 300 of FIG. 3, and a second electronic device 400 of FIG. 4) may obtain user input data for calling an assistant application of the electronic device. The electronic device may obtain pattern information associated with the use of the assistant application through a predictive model based on the user input data and usage history data. The electronic device may generate a prompt based on the pattern information. The prompt may include first prompt information including at least one of a predictive query or a predictive request obtained based on the pattern information. The electronic device may generate a response to the prompt using an artificial intelligence (AI) model (e.g., a generative AI model). The electronic device may provide a message including the response.

[0234] According to one embodiment, the prompt further includes expression method information related to an expression method of the response, wherein the expression method information may include first expression method information for an expression method of a first response associated with the first prompt information.

[0235] According to one embodiment, the prompt further includes second prompt information including at least one of a user request or a user query obtained based on the user input data, wherein the presentation method information includes second presentation method information for a presentation method of a second response associated with the second prompt information, and the second presentation method information may be different from the first presentation method information.

[0236] According to one embodiment, the first expression method information may include information for indirectly expressing the first response associated with the first prompt information, and the second expression method information may include information for directly expressing the second response associated with the second prompt information.

[0237] According to one embodiment, the expression method information may be generated based on at least one of a method of calling the assistant application, a time of calling the assistant application, an acoustic feature associated with the user's speech through the assistant application, or a linguistic feature associated with the user's speech through the assistant application.

[0238] According to one embodiment, the usage history data may include at least one of information about the usage history of the assistant application or information about the usage history of at least one application used through the assistant application.

[0239] According to one embodiment, the user input data includes a user's voice input of the assistant application, wherein the voice input may include a portion corresponding to a name of the assistant application or a portion corresponding to a user request or query.

[0240] In one embodiment, the message may correspond to a welcome message, which is the first message upon invoking the assistant application.

[0241] According to one embodiment, the electronic device may collect at least one piece of relevant information through at least one information collection module based on the pattern information. The at least one piece of relevant information may be used, together with the pattern information, to generate the prompt.

[0242] In one embodiment, the response may include audio information corresponding to the prompt and visual information for providing additional information associated with the audio information.

[0243] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

[0244] The embodiments of this document and the terminology used herein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another component (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0245] The term "module" used in the embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0246] One embodiment of the present document may be implemented as software (e.g., a program (140) of FIG. 1) including one or more instructions stored in a storage medium (e.g., an internal memory (136) of FIG. 1 or an external memory (138) of FIG. 1) readable by a machine (e.g., an electronic device (101) of FIG. 1). For example, a processor (e.g., a processor (120) of FIG. 1) of the machine (e.g., an electronic device (101) of FIG. 1) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0247] According to one embodiment, the method according to one embodiment disclosed in this document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play Store™) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0248] According to one embodiment, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and arranged in other components. According to one embodiment, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to one embodiment, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.< / request> < / request>

Claims

1. In electronic devices, At least one processor comprising a processing circuit; and A memory comprising at least one storage medium for storing instructions, The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: Obtaining user input data that calls the assistant application of the electronic device, Based on the user input data and usage history data associated with the assistant application, pattern information associated with the use of the assistant application is obtained through a prediction model, Generating a prompt based on the pattern information, wherein the prompt includes first prompt information including at least one of a prediction query or a prediction request obtained based on the pattern information, An electronic device that generates a response to the prompt using an AI (artificial intelligence) model and causes the device to provide a message including the response.

2. In paragraph 1, The above prompt further includes expression information related to the expression method of the above response, An electronic device, wherein the above expression method information includes first expression method information for the expression method of the first response associated with the first prompt information.

3. In paragraph 2, The above prompt further includes second prompt information including at least one of a user request or a user query obtained based on the user input data, An electronic device wherein the above expression method information includes second expression method information for the expression method of the second response associated with the second prompt information, wherein the second expression method information is different from the first expression method information.

4. In paragraph 3, An electronic device, wherein the first expression method information includes information for indirectly expressing the first response associated with the first prompt information, and the second expression method information includes information for directly expressing the second response associated with the second prompt information.

5. In any one of paragraphs 1 to 4, An electronic device wherein the expression method information is generated based on at least one of a method of calling the assistant application, a time of calling the assistant application, an acoustic characteristic associated with the user's speech through the assistant application, or a linguistic characteristic associated with the user's speech through the assistant application.

6. In any one of paragraphs 1 to 5, An electronic device wherein the above usage history data includes at least one of information about the usage history of the assistant application or information about the usage history of at least one application used through the assistant application.

7. In any one of paragraphs 1 to 6, An electronic device wherein the user input data includes a user's voice input of the assistant application, wherein the voice input includes at least one of a portion corresponding to a name of the assistant application or a portion corresponding to a user request or query.

8. In any one of paragraphs 1 to 7, The above message corresponds to a welcome message, which is the first message upon calling the assistant application, of the electronic device.

9. In any one of paragraphs 1 to 7, The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: Based on the above pattern information, at least one information collection module causes at least one piece of relevant information to be collected, An electronic device wherein said at least one piece of relevant information is used to generate said prompt together with said pattern information.

10. In any one of paragraphs 1 to 9, An electronic device wherein the response includes audio information corresponding to the prompt and visual information for providing additional information associated with the audio information.

11. In the method of an electronic device, An action of obtaining user input data for calling an assistant application of the electronic device; An operation of obtaining pattern information associated with the use of the assistant application through a prediction model based on the user input data and usage history data associated with the assistant application; An operation of generating a prompt based on the pattern information, the prompt including first prompt information including at least one of a prediction query or a prediction request obtained based on the pattern information; An action to generate a response to the above prompt using an AI (artificial intelligence) model; and A method comprising the action of providing a message including the above response.

12. In paragraph 11, The above prompt further includes expression information related to the expression method of the above response, A method wherein the above expression method information includes first expression method information for the expression method of the first response associated with the first prompt information.

13. In paragraph 12, The above prompt further includes second prompt information including at least one of a user request or a user query obtained based on the user input data, A method wherein the above expression method information includes second expression method information for the expression method of the second response associated with the second prompt information, wherein the second expression method information is different from the first expression method information.

14. In paragraph 13, A method wherein the first expression method information includes information for indirectly expressing the first response associated with the first prompt information, and the second expression method information includes information for directly expressing the second response associated with the second prompt information.

15. In any one of paragraphs 11 to 14, A method wherein the expression method information is generated based on at least one of a method of calling the assistant application, a time of calling the assistant application, an acoustic feature associated with the user's speech through the assistant application, or a linguistic feature associated with the user's speech through the assistant application.

Images