Electronic device, method, and recording medium for supporting interactive service

Abstract

An embodiment of the present disclosure provides an electronic device for supporting an interactive service on the basis of artificial intelligence, an operation method thereof, and a recording medium. An electronic device according to an embodiment may receive a natural language input from a user while executing an interactive service. The electronic device may generate, on the basis of the natural language input, a text prompt including the natural language so as to generate a plurality of tasks with respect to the natural language, and provide the text prompt to an on-device generative AI and / or a generative AI of a server. The electronic device may acquire data including a plurality of tasks on the basis of the text prompt. The electronic device may classify, on the basis of the data, the plurality of tasks according to at least two performing entities among a user of the electronic device, an external device, and / or an application of the electronic device, and allocate at least one task to each performing entity. The electronic device may configure at least one guide and / or at least one menu for executing the at least one task allocated to each performing entity. The electronic device may divide the at least one guide and / or the at least one menu by the at least two performing entities and display same on a display. Various embodiments are possible.

Description

Electronic devices, methods, and recording media supporting interactive services

[0001] An embodiment of the present disclosure provides an electronic device, an operating method thereof, and a recording medium that support an interactive service based on artificial intelligence (AI) (e.g., generative AI).

[0002] With the advancement of digital technology, various types of electronic devices, such as smartphones, tablet PCs (personal computers), laptop computers, desktop computers, digital cameras, and / or wearable devices, are becoming widely used. The hardware and / or software components of these electronic devices are continuously being developed to support and enhance their functionality.

[0003] For example, portable electronic devices (hereinafter referred to as "electronic devices"), such as smartphones, can now be equipped with a variety of functions. Electronic devices include touchscreen-based displays that allow users to easily access various functions, and can display screens for various applications through these displays.

[0004] Recently, with the rapid development of big data and deep learning technologies, artificial intelligence (AI) has been applied to electronic devices. It is also being applied to intelligent personal services that analyze specific data and integrate and utilize information from various fields tailored to the user. For example, users can control electronic devices through voice conversations, and a deep learning-based knowledge base enables them to search for, query, and respond to specific information. Recently, generative AI has been implemented as AI technology evolves. Generative AI can refer to AI technology that generates similar content using existing content, such as text, audio, and / or images. For example, generative AI can refer to AI technology that can generate content (e.g., text, audio, images, and / or video) that responds to a given input.

[0005] For example, recent advancements in technology have enabled the understanding of language (e.g., natural language) through large language models (LLMs) and the summarization or condensation of long sentences. Electronic devices can support interactive services between users and generative AI (or LLM servers), acquire data related to user inquiries through these interactive services, and provide the acquired data to users.

[0006] The above information may be provided as background art to aid in understanding the present disclosure. No claim or determination is made as to whether any of the above-described matters constitute prior art related to the present disclosure.

[0007] In one embodiment of the present disclosure, an electronic device supporting a generative AI-based conversational service, an operating method thereof, and a recording medium are provided.

[0008] In one embodiment of the present disclosure, an electronic device, a method of operation thereof, and a recording medium are provided, which provides a menu (e.g., a function execution menu) that allows a user to easily recognize and execute a task (or action) (e.g., a user task, a device task, and / or an application task) related to the answer content, along with the answer content, while executing an interactive service through a conversational service (or assistant or AI agent) application.

[0009] The technical problems to be achieved in this document are not limited to the technical problems mentioned above, and other technical problems not mentioned can be clearly understood by a person having ordinary skill in the technical field to which the present invention belongs from the description below.

[0010] An electronic device according to an embodiment of the present disclosure may include a display, at least one processor including processing circuitry, and a memory storing instructions (or commands). In one embodiment, the memory may store instructions that, when individually and / or collectively executed by the at least one processor, cause the electronic device to perform operations.

[0011] According to one embodiment, the instructions, when executed by the at least one processor, may cause the electronic device to execute an interactive service. The instructions, when executed by the at least one processor, may cause the electronic device to receive a natural language input from a user while executing the interactive service. The instructions, when executed by the at least one processor, may cause the electronic device to generate a text prompt including the natural language, based on the natural language input, to generate a plurality of tasks for the natural language. The instructions, when executed by the at least one processor, may cause the electronic device to provide the text prompt to an on-device and / or server-based generative artificial intelligence (AI). The instructions, when executed by the at least one processor, may cause the electronic device to obtain data including a plurality of tasks based on the text prompt. The instructions, when executed by the at least one processor, may cause the electronic device to classify the plurality of tasks according to at least two performing entities of a user of the electronic device, an external device, and / or an application of the electronic device based on the data, and to assign at least one task to each performing entity. The instructions, when executed by the at least one processor, may cause the electronic device to configure at least one guide and / or at least one menu for executing the at least one task assigned to each performing entity.The above instructions, when executed by the at least one processor, may cause the electronic device to display the at least one guide and / or the at least one menu on the display, dividing the at least two execution entities.

[0012] An operating method of an electronic device according to an embodiment of the present disclosure may include an operation of executing an interactive service. The operating method may include an operation of receiving a natural language input from a user while executing the interactive service. The operating method may include an operation of generating a text prompt including the natural language based on the natural language input, so as to generate a plurality of tasks for the natural language. The operating method may include an operation of providing the text prompt to a generative artificial intelligence (AI) of an on-device and / or a server. The operating method may include an operation of obtaining data including a plurality of tasks based on the text prompt. The operating method may include an operation of classifying the plurality of tasks based on the data according to at least two performing entities among a user of the electronic device, an external device, and / or an application of the electronic device, and assigning at least one task to each performing entity. The operating method may include an operation of configuring at least one guide and / or at least one menu for executing the at least one task assigned to each performing entity. The above method of operation may include an operation of displaying on a display at least one guide and / or at least one menu by dividing them into at least two performing subjects.

[0013] In order to solve the above-described problem, various embodiments of the present disclosure may include a computer-readable recording medium having recorded thereon a program for executing the method on a processor.

[0014] According to one embodiment, a non-transitory computer-readable recording medium (or storage medium or computer program product) storing one or more programs is described. According to one embodiment, one or more programs may include instructions for performing the following actions: executing an interactive service; receiving a natural language input from a user while executing the interactive service; generating a text prompt including the natural language based on the natural language input to generate a plurality of tasks for the natural language; providing the text prompt to a generative artificial intelligence (AI) of an on-device and / or a server; obtaining data including a plurality of tasks based on the text prompt; classifying the plurality of tasks based on the data according to at least two performing entities of a user of the electronic device, an external device, and / or an application of the electronic device, and allocating at least one task to each performing entity; configuring at least one guide and / or at least one menu for executing the at least one task assigned to each performing entity; and displaying the at least one guide and / or the at least one menu on a display by classifying the at least two performing entities.

[0015] Further scope of the applicability of the present disclosure will become apparent from the detailed description below. However, since various modifications and variations within the spirit and scope of the present disclosure will readily become apparent to those skilled in the art, it should be understood that the detailed description and specific examples, such as preferred embodiments of the present disclosure, are given by way of example only.

[0016] According to an embodiment of the present disclosure, an electronic device, an operation method thereof, and a recording medium, a plurality of tasks can be analyzed based on result data (e.g., natural language-based answers) generated according to a conversation context in an interactive service, and an intuitive interface (e.g., a menu or an action menu) that comprehensively includes a performing entity (e.g., a device, a service / application, and / or a user), a capability of the performing entity, and / or a parameter (e.g., task control information) for each task can be provided.

[0017] In one embodiment, on-device AI services can be effectively provided to users while executing a conversational service. For example, task-related guidance and menus for each guide can be matched and provided together based on result data, allowing users to easily perform tasks simultaneously. For example, task-related conversations (e.g., guidance) and menus for tasks can be provided simultaneously, allowing users to simultaneously check and process tasks and their execution methods while engaging in the conversation. In one embodiment, a new user experience (UX) for conversational services can be provided.

[0018] In addition, various effects may be directly or indirectly realized through this document. The effects obtained through this disclosure are not limited to those mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art to which this disclosure pertains, based on the description below.

[0019] In connection with the description of the drawings, the same or similar reference numerals may be used for the same or similar components.

[0020] FIG. 1 is a block diagram of an electronic device within a network environment according to various embodiments.

[0021] FIG. 2 is a block diagram illustrating an integrated intelligence system according to one embodiment.

[0022] FIG. 3 is a block diagram illustrating a generative artificial intelligence system according to one embodiment.

[0023] FIG. 4 is a diagram schematically illustrating the configuration of an electronic device according to one embodiment.

[0024] FIG. 5 is a flowchart illustrating a method of operating an electronic device according to one embodiment.

[0025] FIG. 6 is a diagram illustrating an example of a general interface supporting interactive services in an electronic device according to one embodiment.

[0026] FIG. 7 is a diagram illustrating an example of an interface that supports interactive services in an electronic device according to one embodiment.

[0027] FIG. 8 is a diagram illustrating an example of an interface that supports interactive services in an electronic device according to one embodiment.

[0028] FIG. 9 is a diagram illustrating an example of a menu provided by an electronic device according to one embodiment.

[0029] FIG. 10 is a drawing illustrating an example of a menu provided by an electronic device according to one embodiment.

[0030] FIG. 11 is a diagram illustrating an example of a menu provided by an electronic device according to one embodiment.

[0031] FIG. 12 is a diagram illustrating an example of a menu provided by an electronic device according to one embodiment.

[0032] FIG. 13 is a diagram illustrating an example of providing a menu in an electronic device according to one embodiment.

[0033] FIG. 14 is a diagram illustrating an example of providing a menu in an electronic device according to one embodiment.

[0034] FIG. 15 is a diagram illustrating an example of an operation for supporting an interactive service in an electronic device according to one embodiment.

[0035] FIG. 16 is a diagram illustrating an example of an operation for supporting an interactive service in an electronic device according to one embodiment.

[0036] FIG. 17 is a diagram illustrating an example of an operation for supporting an interactive service in an electronic device according to one embodiment.

[0037] FIG. 18 is a diagram illustrating an example of an operation of executing a routine in an electronic device according to one embodiment.

[0038] FIG. 19 is a diagram illustrating an example of an operation of executing a routine in an electronic device according to one embodiment.

[0039] FIG. 20 is a diagram illustrating an example of an operation of executing a routine in an electronic device according to one embodiment.

[0040] FIGS. 21a, 21b, 21c, 21d, and 21e are diagrams illustrating examples of operations for supporting interactive services in an electronic device according to one embodiment.

[0041] FIG. 22 is a flowchart illustrating a method of operating an electronic device according to one embodiment.

[0042] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. In connection with the description of the drawings, the same or similar reference numerals may be used for identical or similar components. Furthermore, in the drawings and related descriptions, descriptions of well-known functions and configurations may be omitted for clarity and conciseness.

[0043] FIG. 1 is a block diagram of an electronic device (101) within a network environment (100) according to various embodiments.

[0044] Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of an electronic device (104) or a server (108) via a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0045] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or operations. According to one embodiment, as at least a part of the data processing or operations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit (CPU) or an application processor (AP)) or an auxiliary processor (123) (e.g., a graphic processing unit (GPU), a neural processing unit (NPU), an image signal processor (ISP), a sensor hub processor, or a communication processor (CP)) that can operate independently or together with the main processor (121). For example, when the electronic device (101) includes the main processor (121) and the auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a given function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0046] The auxiliary processor (123) may control at least a part of functions or states related to at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0047] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0048] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (OS) (142), middleware (144), or an application (146).

[0049] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0050] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0051] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. In one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0052] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0053] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0054] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, a secure digital (SD) card interface, or an audio interface.

[0055] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0056] A haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. In one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0057] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0058] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least a part of a power management integrated circuit (PMIC).

[0059] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0060] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or a wide area network (WAN))). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0061] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). NR access technology can support high-speed transmission of high-capacity data (eMBB, enhanced mobile broadband), minimizing terminal power and connecting multiple terminals (mMTC, massive machine type communications), or high reliability and low latency communications (URLLC, ultra-reliable and low-latency communications). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0062] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas by, for example, the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0063] According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high frequency band.

[0064] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0065] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server using machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0066] FIG. 2 is a block diagram illustrating an integrated intelligence system according to one embodiment.

[0067] Referring to FIG. 2, an integrated intelligent system of one embodiment may include an electronic device (201) (e.g., the electronic device (101) of FIG. 1), an intelligent server (300), and a service server (399).

[0068] According to the illustrated embodiment, the electronic device (201) may include a communication interface (210), an input / output (I / O) interface (220), a processor (230), and / or a memory (240). The components listed above may be operatively or electrically connected to each other. For example, the electronic device (201) may include at least some of the components of the electronic device (101) of FIG. 1.

[0069] The communication interface (210) can be connected to an external device (e.g., an intelligent server (300) and / or a service server (399)) via a network (299) (e.g., any network including a cellular network and / or a wireless local area network (WLAN)) to transmit and receive data. For example, the communication interface (210) can correspond to the CP and / or communication circuit of FIG. 1. The I / O interface (220) can receive user input, process received user input, and / or output a result processed by the processor (230) using an input / output device (not shown) (e.g., a microphone, a speaker, and / or a display (e.g., a display of FIG. 1).

[0070] The processor (230) may be operatively or electrically connected to the communication interface (210), the I / O interface (220), and / or the memory (240) (e.g., the memory of FIG. 1) to perform a designated operation. For example, the processor (230) may correspond to the processor (120) of FIG. 1. The processor (230) may execute a program (or one or more instructions) stored in the memory (240) to perform a designated operation. For example, the processor (230) may receive a user's voice input (e.g., a user's speech) through the I / O interface (220) or from an external electronic device. The processor (230) may transmit the voice input received through the communication interface (210) to the intelligent server (300). For example, the processor (230) may include one or more processors.

[0071] The processor (230) may receive a result corresponding to the voice input from the intelligent server (300). For example, the processor (230) may receive a plan corresponding to the voice input and / or a result calculated using the plan from the intelligent server (300). For example, the plan may include, but is not limited to, information regarding a plurality of sequential operations to be executed by the first electronic device (201) and / or another electronic device in relation to the voice input. The processor (230) may receive a request from the intelligent server (300) to obtain information (e.g., entities, slots, and / or parameters) necessary to generate a plan corresponding to the voice input. The processor (230) may transmit the necessary information to the intelligent server (300) in response to the request.

[0072] The processor (230) may visually, tactilely, and / or audibly output the results of executing the operations specified according to the plan via the I / O interface (220). For example, the processor (230) may sequentially display the execution results of multiple operations on the display. As an example, the processor (230) may display only the execution results of executing multiple operations (e.g., the execution results of one of the multiple operations or the last operation) on the display.

[0073] The processor (230) can recognize voice input. For example, the processor (230) can execute an intelligent app (or a voice recognition app) to process the voice input in response to a specified voice input (e.g., "Wake up!"). The processor (230) can provide a voice recognition service through the intelligent app. The processor (230) can transmit the voice input to the intelligent server (300) through the intelligent app and receive a result corresponding to the voice input from the intelligent server (300).

[0074] An intelligent server (300) of one embodiment can receive a user's voice input from an electronic device (201) via a network (299). The intelligent server (300) can convert audio data corresponding to the received voice input into text data. The intelligent server (300) can generate at least one plan for performing a task corresponding to the user's voice input based on the text data. The intelligent server (300) can transmit the generated plan or a result according to the generated plan to the electronic device (201) via the network (299).

[0075] An intelligent server (300) of one embodiment may include a front end (310), a natural language platform (320), a capsule database (330), an execution engine (340), and / or an end user interface (350).

[0076] The front end (310) can receive a voice input received by the electronic device (201) from the electronic device (201). The front end (310) can transmit a response corresponding to the voice input to the electronic device (201).

[0077] The natural language platform (320) may include an automatic speech recognition (ASR) module (321), a natural language understanding (NLU) module (323), a planner module (325), a natural language generator (NLG) module (327), and / or a text-to-speech (TTS) module (329).

[0078] The automatic speech recognition module (321) can convert the voice input received from the electronic device (201) into text data. The natural language understanding module (323) can identify the user's intent and / or parameters (e.g., entities and / or slots) based on the text data of the voice input. The user's intent corresponds to the voice input and may include information indicating an action (or function) that the user wishes to perform using the device. The slot may be detailed information related to the user's intent. The slot may be acquired based on a domain corresponding to the utterance. The slot may be variable information required to perform the action. In one embodiment, the variable information constituting the slot may include a named entity.

[0079] The planner module (325) can generate a plan using the intent and / or parameters determined by the natural language understanding module (323). For example, the planner module (325) can determine at least one domain necessary to perform a task based on the determined intent. The planner module (325) can determine a plurality of operations included in each of the at least one domain determined based on the intent. The domain may correspond to a category (or service) associated with an operation (or function) that the user wishes to perform using the device. The domain may be classified according to a service (e.g., an app) related to the text. The domain may be related to the user's intent corresponding to the text. The domain may be classified according to, for example, the type of application that received the voice input and / or the type of service to be provided based on the voice input, but is not limited thereto. In one example, the determination of the domain may be performed by another module (e.g., the natural language understanding module (323)). The planner module (325) can determine parameters required to execute a plurality of determined actions or result values ​​output by the execution of the plurality of actions. The parameters and result values ​​can be defined as concepts of a specified format (or class). For example, the plan can include a plurality of actions and / or a plurality of concepts determined by the user's intention. The planner module (325) can determine the relationship between the plurality of actions and / or the plurality of concepts in a step-by-step (or hierarchical) manner. For example, the planner module (325) can identify the execution order of the plurality of actions (e.g., the plurality of actions determined based on the user's intention) based on the plurality of concepts (e.g., parameters required to execute the plurality of actions and results output by the execution of the plurality of actions). The planner module (325) can generate a plan including association information (e.g., ontology) between the plurality of actions and the plurality of concepts.The planner module (325) can create a plan using information (e.g., at least one capsule) stored in a capsule database (330) in which a set of relationships between concepts and actions is stored.

[0080] The planner module (325) can generate a plan based on an artificial intelligence (AI) system. For example, the AI ​​system can include one or more electronic devices and / or one or more processing circuits to execute a rule-based system, a neural network-based system (e.g., a feedforward neural network (FNN), a recurrent neural network (RNN)), or a combination thereof. The AI ​​system described above is exemplary, and the AI ​​system can be an AI system based on any machine learning-based model. The planner module (325) can select a plan corresponding to a user request from a set of predefined plans, or generate a plan in real time in response to a user request.

[0081] The natural language generation module (327) can convert specified information into text format. The information converted into text format may be in the form of natural language speech. The text-to-speech conversion module (329) can convert information in text format into information in speech format.

[0082] The capsule database (330) can store information on the relationship between multiple concepts and actions corresponding to multiple domains (e.g., applications). The capsule database (330) can store at least one capsule (e.g., capsule (331) and / or capsule (333)) in the form of a concept action network (CAN). For example, the capsule database (330) can store actions for processing tasks corresponding to a user's voice input and parameters required for the actions in the form of a CAN. A capsule can include multiple action objects (or action information) and / or concept objects (or concept information) included in a plan. For example, capsules (331, 333) can be created for each domain and stored in the capsule database (330), but are not limited thereto.

[0083] The execution engine (340) can produce results using the generated plan. The end user interface (350) can transmit the produced results to the electronic device (201).

[0084] According to one embodiment, some functions (e.g., the natural language platform (320)) or all functions of the intelligent server (300) may be implemented in the electronic device (201). For example, the electronic device (201) may execute one or more programs including the natural language platform (e.g., the natural language platform (320) of FIG. 2) separately from the intelligent server (300). For example, the electronic device (201) may directly perform at least some of the operations of the natural language platform (320) of the intelligent server (300) (e.g., the automatic speech recognition module (321), the natural language understanding module (323), the planner module (325), the natural language generation module (327), and / or the text-to-speech module (329)).

[0085] In one embodiment, a service server (399) may provide a service (e.g., food ordering or hotel reservation) designated to an electronic device (201). The service server (399) may be a server operated by a different operator than the intelligent server (300). The service server (399) may communicate with the intelligent server (300) and / or the electronic device (201) via a network (299). The service server (399) may communicate with the intelligent server (300) via a separate connection (not shown). The service server (399) may provide the intelligent server (300) with information for generating a plan corresponding to a voice input received in the electronic device (201) (e.g., operation information and / or concept information for providing a designated service). The provided information may be stored in a capsule database (330). The service server (399) may provide the intelligent server (300) with result information according to the plan received from the electronic device (201).

[0086] FIG. 3 is a block diagram illustrating a generative artificial intelligence system according to one embodiment.

[0087] Referring to FIG. 3, a generative artificial intelligence system (e.g., an intelligent server, server (108) of FIG. 1) according to one embodiment may include a user interface (260), a database (265), an applications / service component (270), an AI framework (280), and a generative AI model (290).

[0088] The user interface (260) can receive a user query. The user query can be in the form of natural language, images, and videos. Additionally, context information can be transmitted along with the user query. As another example, the user query can also be a non-natural language input that does not generate natural language, such as a design request or modification. Furthermore, the user interface (260) can also be in the form of a mixture of natural language, images, sounds, and context information as described above. Furthermore, the user interface (260) can output the results of the generative artificial intelligence system to the user. The output can be in the form of natural language or specific content, and can also be provided in the form of an action requested by the user.

[0089] The AI ​​framework (280) can receive a user query and coordinate and control each component necessary to carry out the user's intent. The AI ​​framework (280) may include a prompt design component (281), an application and plug-in management component (APIs / Plugins Management component) (283), and an output modification component (285).

[0090] A user query or action entered in the user interface (260) can be transmitted to a prompt design component (281). The prompt design component (281) can be used to generate prompts suitable for input into a large language model (LLM) or a large multimodal model. The prompt design component (281) can be an AI component that uses a machine learning algorithm or a neural network to develop better prompts over time. The prompt design component (281) can access a knowledge component containing user preference data, a prompt library, and prompt examples to generate prompts and transmit them to the large language model (LLM) or the large multimodal model (LMM).

[0091] The application and plugin management component (283) can communicate with external information when a request for additional information is made when user input is passed as input to the generative model. The application and plugin management component (283) establishes a channel for communication with the AI ​​Interface externally via an application programming interface (API), thereby enabling access to various data sources. Furthermore, the application and plugin management component (283) can request an action via the API to ultimately perform a user query, rather than an intermediate result, if the application or service needs to perform that action. Information obtained from external sources can be passed as input to the generative model along with user input.

[0092] The output modification component (285) can fine-tune the output from the generative model. For example, the output modification component (285) can verify that the content generated through a language model (LLM) or a large-scale multi-modal model (LMM) is not irrelevant, does not contain biased content, or does not contain harmful content. In addition, the output modification component (285) can determine to what extent the content matches the user's desired result and can proceed with additional processing if necessary. Additionally, the output modification component (285) can configure and provide the user with hints to avoid undesired output.

[0093] A generative AI model (290) generally refers to an artificial intelligence neural network that creates new types of data based on user input information. Representative models that generate images include the generative adversarial network (GAN) and the variational auto encoder (VAE). Recently, diffusion-based generative models that use VAE and Transformer structures are called generative models. In addition, language models are models trained to statistically output the most appropriate output based on input values, and representative examples include models such as CHAT-GPT 3 and CHAT-GPT 4. In addition, since they can recognize various types of data input, such as text, images, and voice, and generate new data corresponding to them, they are called LMMs (large multimodal models).

[0094] FIG. 4 is a diagram schematically illustrating the configuration of an electronic device according to one embodiment of the present disclosure.

[0095] Referring to FIG. 4, an electronic device (101) according to one embodiment of the present disclosure may include a display (490) (e.g., a display module (160) of FIG. 1 or an I / O interface (220) of FIG. 2), a memory (130) (e.g., a memory (130, 240) of FIG. 1 or 2), a communication circuit (495) (e.g., a communication module (190) or a communication interface (210) of FIG. 1), and / or a processor (120) (e.g., a processor (120, 230) of FIG. 1 or 2).

[0096] According to one embodiment, the electronic device (101) may include all or at least a portion of the components of the electronic device (101, 201) as described in the description with reference to FIG. 1 or FIG. 2. For example, in various embodiments of the present document, some of the illustrated components may be omitted or replaced. The electronic device (101) may include at least a portion of the components and / or functions of the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2. At least some of the respective components of the illustrated (or not illustrated) electronic devices (101, 201) may be operatively, functionally and / or electrically connected to each other.

[0097] According to one embodiment, the display (490) may include a configuration identical or similar to the display module (160) of FIG. 1. According to one embodiment, the display (490) may display various images provided from the processor (120). According to one embodiment, the display (490) may, under the control of the processor (120), visually provide an application being executed (e.g., the application (146) of FIG. 1) and various screens related to its use (e.g., a contents screen, an application (e.g., an assistant application, an interactive service application, or an AI agent) execution screen, a menu screen, and / or a function execution screen).

[0098] According to one embodiment, the display (490) may be combined with a touch sensor, a pressure sensor capable of measuring the intensity of a touch, and / or a touch panel (e.g., a digitizer) that detects a magnetic stylus pen. According to one embodiment, the display (490) may detect a touch input, an air gesture input, and / or a hovering input (or a proximity input) by measuring a change in a signal (e.g., voltage, light intensity, resistance, electromagnetic signal, and / or charge) for a specific location of the display (490) based on the touch sensor, the pressure sensor, and / or the touch panel. For example, the display (490) may include a touchscreen that detects a touch and / or a proximity touch (or a hovering) input using a part of a user's body (e.g., a finger) or an input device (e.g., a stylus pen). The display (490) may include at least some of the configuration and / or functions of the display module (160) of FIG. 1 and / or the I / O interface (220) of FIG. 2.

[0099] In one embodiment, the display (490) may include, but is not limited to, a liquid crystal display (LCD), a light-emitting diode (LED), an organic light-emitting diode (OLED) display, and / or an active matrix OLED (AMOLED) display, a micro electro mechanical systems (MEMS) display, or an electronic paper display. In one embodiment, the display (490) may include a flexible display.

[0100] According to one embodiment, the memory (130) includes at least a portion of the configuration and / or function of the memory (130) of FIG. 1 and / or the memory (240) of FIG. 2, and may store software (e.g., the program (140) of FIG. 1). The memory (130) may store various applications (e.g., the application (146) of FIG. 1) and program modules supporting intelligent services (e.g., a client module or a generative AI model).

[0101] According to one embodiment, the memory (130) may store various data used by at least one component (e.g., processor (120)) of the electronic device (101). In one embodiment, the data may include, for example, software (e.g., program (140) of FIG. 1), and input data or output data for commands related to the software.

[0102] According to one embodiment, the memory (130) may include volatile memory (e.g., volatile memory (132) of FIG. 1) or non-volatile memory (134) (e.g., non-volatile memory (134) of FIG. 1). According to one embodiment, the memory (130) may store instructions or data received from the processor (120) in the volatile memory (132), and may store result data of instructions or data stored in the volatile memory (132) being processed by the processor (120) in the non-volatile memory (134).

[0103] In one embodiment, the data may include various data (e.g., learning data, prompt data, context, and / or learning models) to support the electronic device (101) in generating artificial intelligence-based data (e.g., text data, menu data, task data, and / or action data) (e.g., a response (e.g., text or guide) to a user query (or inquiry) and a menu that can execute a task (or action) related to the content). In one embodiment, a task may correspond to something that a performer needs to perform (e.g., a task, an action, an experience, a service). In one embodiment, a guide may correspond to a final display result. In one embodiment, an action may correspond to an action that a performer performs.

[0104] In one embodiment, a task may include a user task, a device task, and / or an application task depending on a performing entity (e.g., a user of the electronic device (101), an external device (or devices), and / or an application of the electronic device (101). A user task may include a task performed by a user. A device task may include a task performed in conjunction with another electronic device (e.g., a device of things or an IoT device) in the vicinity that is interoperable (or connected) with the electronic device (101). An application task may include a task performed using an application set (or installed or installable) in the electronic device (101). In one embodiment, the data may include information regarding various settings for supporting the electronic device (101) to control an operation of editing and / or generating artificial intelligence-based data.

[0105] In one embodiment, the data may include various learning data and / or parameters acquired based on the user's learning through interaction with the user. In one embodiment, the data may include various schemas (or algorithms, models, networks, or functions) for supporting artificial intelligence-based conversational service operations.

[0106] For example, a scheme for supporting artificial intelligence-based conversational service operation in an electronic device (101) may include a neural network. In one embodiment, the neural network may include a neural network model based on at least one of an artificial neural network (ANN), a convolution neural network (CNN), a region with convolution neural network (R-CNN), a region proposal network (RPN), a recurrent neural network (RNN), a stacking-based deep neural network (S-DNN), a state-space dynamic neural network (S-SDNN), a deconvolution network, a deep belief network (DBN), a restricted Boltzman machine (RBM), a long short-term memory (LSTM) network, a classification network, a plain residual network, a dense network, a hierarchical pyramid network, and / or a fully convolutional network. According to one embodiment, the type of the neural network model is not limited to the examples described above.

[0107] According to one embodiment, the memory (130) may store instructions that, when executed, cause the processor (120) to operate. The memory (130) may store instructions that, when individually and / or collectively executed by the processor (120), cause the electronic device (101) to perform operations.

[0108] According to one embodiment, the memory (130) may store instructions that, when individually and / or collectively executed by the processor (120), cause the electronic device (101) to execute an interactive service, receive a natural language input from a user while executing the interactive service, generate a text prompt including natural language to generate a plurality of tasks for the natural language input, provide the text prompt to a generative artificial intelligence (AI) on-device and / or server, obtain data including a plurality of tasks based on the text prompt, classify the plurality of tasks based on the data according to at least two performing entities among a user of the electronic device, an external device, and / or an application of the electronic device, assign at least one task to each performing entity, configure at least one guide and / or at least one menu for executing the at least one task assigned to each performing entity, and display the at least one guide and / or at least one menu on a display by classifying the at least two performing entities.

[0109] For example, instructions may be stored as software (e.g., program (140) of FIG. 1) on memory (130) and executable by processor (120). For example, instructions may include control commands such as arithmetic and logical operations, data movement, and / or input / output that may be recognized by processor (120). According to one embodiment, the software may include various applications (e.g., application (146) of FIG. 1) that may provide various functions (or services) (e.g., interactive service function, routine function, call function, message function, messenger function, e-mail function, SNS (social networking service) function, search function, media (e.g., video and / or music) playback function, game function, and / or wireless communication function) in electronic device (101).

[0110] According to one embodiment, the communication circuit (495) may support the establishment of a designated wireless communication channel (e.g., short-range communication such as Bluetooth communication and / or BLE communication) and the performance of communication through the established wireless communication channel. For example, the communication circuit (495) may perform designated communication (e.g., Bluetooth communication and / or BLE communication) with an external device. According to one embodiment, the communication circuit (495) may support wireless communication with an external device using cellular wireless communication (e.g., 4G LTE, 5G NR) and / or short-range wireless communication (e.g., Wi-Fi). For example, the electronic device (101) may use the communication circuit (495) to communicate with an external server (e.g., a generative artificial intelligence server or an LLM server) that provides an artificial intelligence-based function (e.g., a conversational service or an assistant service or an AI agent) through a network. According to one embodiment, the communication circuit (495) can transmit data (e.g., natural language-based prompt) generated in the electronic device (101) to an external server, and can receive data (e.g., response content (e.g., text, image, and / or video)) transmitted from the external server. According to one embodiment, the communication circuit (495) can include at least some of the configurations and / or functions of the communication module (190) of FIG. 1 and / or the communication interface (210) of FIG. 2.

[0111] According to one embodiment, the processor (120) may perform an application layer processing function requested by a user of the electronic device (101). According to one embodiment, the processor (120) may provide control and commands of functions for various blocks of the electronic device (101). According to one embodiment, the processor (120) may perform operations or data processing related to control and / or communication of each component of the electronic device (101). For example, the processor (120) may include at least some of the configurations and / or functions of the processor (120) of FIG. 1. According to one embodiment, the processor (120) may be operatively connected to the components of the electronic device (101). According to one embodiment, the processor (120) may load commands or data received from other components of the electronic device (101) into the memory (130), process the commands or data stored in the memory (130), and store result data.

[0112] According to one embodiment, the processor (120) may include at least one processor including processing circuitry and / or executable program elements. According to one embodiment, the processor (120) may control (or process) the overall operation related to the function of the electronic device (101) based on the processing circuitry and / or the executable program elements (e.g., generating a prompt (e.g., a text prompt) based on natural language input by a user (e.g., voice-based natural language based on the user's utterance and / or text-based natural language based on the user's text input), and supporting an interactive service (e.g., a function of providing an answer to a user query (or inquiry)) through the generated prompt.

[0113] In one embodiment, the processor (120) can execute an interactive service. In one embodiment, the processor (120) can receive a natural language input from a user while executing the interactive service. In one embodiment, the processor (120) can generate a prompt based on the natural language input. In one embodiment, the processor (120) can generate a text prompt including natural language to generate a plurality of tasks for the natural language based on the natural language input. In one embodiment, the processor (120) can provide the prompt (e.g., the text prompt) to a generative artificial intelligence (AI) on-device and / or on a server.

[0114] According to one embodiment, the processor (120) may obtain data including a plurality of tasks based on a prompt. According to one embodiment, the processor (120) may classify the plurality of tasks based on the data according to at least two performing entities among a user of the electronic device, an external device, and / or an application of the electronic device. According to one embodiment, the processor (120) may assign at least one task to each performing entity. According to one embodiment, the processor (120) may configure at least one guide and / or at least one menu for executing at least one task assigned to each performing entity. According to one embodiment, the processor (120) may display at least one guide and / or at least one menu on a display by classifying at least two performing entities.

[0115] In one embodiment, the processor (120) can determine (or identify) information related to a conversation topic, task, user of the electronic device, external device, and / or application of the electronic device in natural language. In one embodiment, the processor (120) can generate a prompt (e.g., a text prompt) based on the natural language and information.

[0116] According to one embodiment, the processor (120) may obtain data from the generative AI, based on a prompt (e.g., a text prompt), including information about a title related to a conversation topic, a plurality of tasks, and an executing entity related to the plurality of tasks.

[0117] In one embodiment, at least one menu may include spatial information and route guidance information regarding the execution of the execution entity. In one embodiment, the spatial information and route guidance information may be displayed by being arranged on a spatial diagram provided by the electronic device.

[0118] According to one embodiment, the processor (120) may obtain a spatial diagram defined in relation to the performing subject. According to one embodiment, the processor (120) may display information related to the activity path of the performing subject based on the spatial diagram.

[0119] According to one embodiment, the processor (120) may obtain result data related to the user's task in relation to the prompt. According to one embodiment, the processor (120) may classify guides (or conversations) for each user's task based on the result data. According to one embodiment, the processor (120) may identify actions corresponding to each classified guide. According to one embodiment, the processor (120) may configure (or generate) at least one menu for executing the action. According to one embodiment, the processor (120) may generate output data to be provided to the user based on the guide and menu. According to one embodiment, the processor (120) may display output data including the guide and menu in a time-series manner on a display in relation to natural language input.

[0120] According to one embodiment, the processor (120) may determine, based on natural language input, a conversation topic, a task, an action performer for each task, a device related to the action performer, and / or an application related to the action performer. According to one embodiment, the processor (120) may generate a prompt based on the natural language and the judgment result. According to one embodiment, the processor (120) may obtain result data in relation to the prompt.

[0121] According to one embodiment, the processor (120) may generate a prompt based on natural language input. According to one embodiment, the processor (120) may obtain result data including information about a title related to a conversation topic in natural language, a task, a guide related to the task, an action performer for each task, a device related to the action performer, and / or an application related to the action performer, in relation to the prompt.

[0122] In one embodiment, the processor (120) can extract tasks based on the result data. In one embodiment, the processor (120) can classify guides by task. In one embodiment, the processor (120) can determine an execution entity to perform a task according to the guide. In one embodiment, the processor (120) can identify a menu that allows the execution entity to execute a defined action according to the task. In one embodiment, the processor (120) can generate output data based on the guide and the menu.

[0123] In one embodiment, the menu may include the capabilities of the agent performing the task and parameters for controlling actions based on the capabilities. In one embodiment, the capabilities may include the functions (or services or operations) provided for the task. In one embodiment, the parameters may include setting values ​​related to the capabilities.

[0124] In one embodiment, the processor (120) may extract capabilities and parameters based on the result data. In one embodiment, the processor (120) may set capabilities to be performed for a task and parameters for operating functions based on the capabilities. In one embodiment, the processor (120) may provide at least one action menu including capabilities and / or parameters.

[0125] According to one embodiment, the processor (120) may extract a title related to the conversation topic based on the result data. According to one embodiment, the processor (120) may generate a title menu including the title. According to one embodiment, the processor (120) may provide the title menu at the top. According to one embodiment, the processor (120) may provide at least one action menu at the bottom of the title menu in a hierarchical structure that is divided by task and has hierarchical connections.

[0126] According to one embodiment, the detailed operation of the processor (120) of the electronic device (101) is described with reference to the drawings described below.

[0127] According to one embodiment, the processors (120) can operate individually and / or collectively.

[0128] According to one embodiment, the processor (120) may include an application processor (AP) and / or a communication processor (CP). According to one embodiment, the communication processor may be included in and operate in the communication circuit (495). According to one embodiment, the processor (120) may be an application processor. For example, the processor (120) may be a system semiconductor that is responsible for various functions (e.g., calculation and multimedia driving functions) of the electronic device (101). According to one embodiment, the processor (120) may be configured in the form of a system-on-chip (SoC), and may include a technology-intensive semiconductor chip (e.g., an application processor) that integrates multiple semiconductor technologies into one and implements system blocks into a single chip.

[0129] According to one embodiment, the system blocks of the processor (120) may include components such as a graphics processing unit (GPU) (410), an image signal processor (ISP) (420), a central processing unit (CPU) (430), a neural processing unit (NPU) (440), a digital signal processor (DSP) (450), a modem (460), connectivity (470), and / or security (480), as illustrated in FIG. 4.

[0130] In one embodiment, the GPU (410) may be responsible for graphics processing. In one embodiment, the GPU (410) may receive commands from the CPU (430) and perform graphics processing to express the shape, position, color, shading, movement, and / or texture of objects (or objects) on the display.

[0131] In one embodiment, the ISP (420) may be responsible for image processing and correction of images and videos. In one embodiment, the ISP (420) may correct raw data (e.g., raw data) transmitted from an image sensor of a camera (e.g., the camera module (180) of FIG. 1) to generate an image in a form more preferred by the user. In one embodiment, the ISP (420) may perform post-processing, such as adjusting partial brightness of an image and emphasizing detailed parts. For example, the ISP (420) may independently perform a process of tuning and correcting the image quality of an image acquired through a camera to generate a result preferred by the user.

[0132] According to one embodiment, the ISP (420) may support artificial intelligence (AI)-based image processing technology. According to one embodiment, the ISP (420) may support scene segmentation (e.g., image segmentation) technology that recognizes and / or classifies parts of a scene being captured in conjunction with the NPU (440). For example, the ISP (420) may include a function that applies different parameters to objects such as the sky, bushes, and / or skin and processes them. According to one embodiment, the ISP (420) may detect and display a human face during image capture using the AI ​​function, or adjust the brightness, focus, and / or color of the image using the coordinates and information of the face.

[0133] According to one embodiment, the CPU (430) can perform operations corresponding to the processor (120). According to one embodiment, the CPU (430) can decode a user's command, perform arithmetic and logical operations, and / or data processing operations. For example, the CPU (430) can be responsible for functions such as memory, interpretation, calculation, and control. According to one embodiment, the CPU (430) can control the overall function of the electronic device (101). For example, the CPU (430) can execute all software (e.g., application (146) of FIG. 1) of the electronic device (101) on an operating system (OS) and control hardware devices. According to one embodiment, the CPU (430) can execute an application and control the overall operation of the processor (120) to perform neural network-based tasks required according to the execution of the application.

[0134] According to one embodiment, the CPU (430) may store instructions or data in volatile memory (e.g., volatile memory (132) of FIG. 1) of the memory (130), process the instructions or data stored in the volatile memory, and store resultant data in nonvolatile memory (e.g., nonvolatile memory (134) of FIG. 1) of the memory (130) as at least part of data processing or calculation.

[0135] According to one embodiment, the CPU (430) may include a single processor core or multiple processor cores (multi-core). According to one embodiment, the CPU (430) may be a programmable processor that stores executable instructions (e.g., instructions capable of performing operations of the CPU (430)) and executes the instructions.

[0136] According to one embodiment, the CPU (430) can operate in a multi-domain environment. According to one embodiment, the CPU (430) can operate in a multi-domain environment of a normal world (e.g., a non-secure world, a framework, or a non-secure environment) and a secure world (e.g., a secure framework or a secure environment). In one embodiment, a domain of the secure world can include one or more domains (e.g., a trusted OS, a trust zone, and / or a virtualization framework).

[0137] According to one embodiment, the NPU (440) can perform processing optimized for artificial intelligence deep-learning algorithms. According to one embodiment, the NPU (440) is a processor optimized for deep-learning algorithm operations (e.g., artificial intelligence operations) and can process big data quickly and efficiently like a human neural network. For example, the NPU (440) can be mainly used for artificial intelligence operations. According to one embodiment, the NPU (440) can recognize objects, environments, and / or people in the background when taking a video through a camera and automatically adjust the focus, automatically switch the shooting mode of the camera module (180) to food mode when taking a picture of food, and / or perform processing to erase only unnecessary subjects from the captured results. According to one embodiment, the NPU (440) can perform processing to generate an answer based on given information (e.g., natural language).

[0138] According to one embodiment, the electronic device (101) can support integrated machine learning processing by interacting with all processors such as the GPU (410), the ISP (420), the CPU (430), and the NPU (440).

[0139] In one embodiment, the DSP (450) may represent an integrated circuit that facilitates rapid processing of digital signals. In one embodiment, the DSP (450) may perform the function of converting analog signals into digital signals and performing high-speed processing.

[0140] According to one embodiment, the modem (460) may perform operations that enable the use of various communication functions in the electronic device (101). For example, the modem (460) may support communications such as telephone and data transmission and reception by exchanging signals with a base station. According to one embodiment, the modem (460) may include an integrated modem (e.g., a cellular modem, an LTE modem, a 5G modem, a 5G-Advanced modem, and a 6G modem) that supports communication technologies such as long term evolution (LTE) and 2G to 5G. According to one embodiment, the modem (460) may include an artificial intelligence modem that applies an artificial intelligence algorithm.

[0141] In one embodiment, the connectivity (470) may support wireless data transmission based on IEEE 802.11. In one embodiment, the connectivity (470) may support communication services based on IEEE 802.11 (e.g., Wi-Fi) and / or 802.15 (e.g., Bluetooth, ZigBee, UWB). For example, the connectivity (470) may support communication services targeting an unspecified number of people in a localized area, such as indoors, using an unlicensed band.

[0142] According to one embodiment, security (480) can provide an independent security execution environment between data or services stored in the electronic device (101). According to one embodiment, security (480) can perform an operation to prevent external hacking through software and hardware security during the process of user authentication when providing services such as biometrics, mobile identification, and / or payment of the electronic device (101). For example, security (480) can provide an independent security execution environment in device security for reinforcing the security of the electronic device (101) itself and in security services based on user information such as mobile identification, payment, and car keys in the electronic device (101).

[0143] According to one embodiment, the operations performed by the processor (120) may be implemented by executing instructions stored in a recording medium (or a computer program product or storage medium). For example, the recording medium may include a non-transitory computer-readable recording medium having recorded thereon a program for executing various operations performed by the processor (120).

[0144] The embodiments described in the present disclosure may be implemented in a computer-readable recording medium using software, hardware, or a combination thereof. In a hardware implementation, the operations described in one embodiment may be implemented using at least one of application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, micro-controllers, microprocessors, and / or other electrical units for performing functions.

[0145] In one embodiment, a computer-readable recording medium (or computer program product or storage medium) is provided, which records a program for causing an electronic device (101) to perform (or execute) various operations.

[0146] The above operations may include: an operation of executing an interactive service; an operation of receiving a natural language input from a user while executing the interactive service; an operation of generating a text prompt including natural language to generate a plurality of tasks for the natural language based on the natural language input; an operation of providing the text prompt to a generative artificial intelligence (AI) of an on-device and / or a server; an operation of acquiring data including a plurality of tasks based on the text prompt; an operation of classifying the plurality of tasks based on the data according to at least two performing entities among a user of the electronic device, an external device, and / or an application of the electronic device; an operation of assigning at least one task to each performing entity; an operation of configuring at least one guide and / or at least one menu for executing the at least one task assigned to each performing entity; and an operation of displaying at least one guide and / or at least one menu by dividing the at least two performing entities into the at least two performing entities.

[0147] The above operations may include an operation of executing an interactive service, an operation of receiving a natural language input from a user while executing the interactive service, an operation of generating a prompt based on the natural language input, an operation of obtaining result data related to a user's task in relation to the prompt, an operation of classifying guides for each task of the user based on the result data, an operation of identifying actions corresponding to each of the classified guides, an operation of configuring at least one menu for executing an action, an operation of generating output data to be provided to the user based on the guide and the menu, and an operation of displaying output data including the guide and the menu in a time-series manner in relation to the natural language input.

[0148] An electronic device (e.g., the electronic device (101, 201) of FIGS. 1, 2, and / or 4) according to one embodiment of the present disclosure may include a display (e.g., the display module (160) of FIG. 1 or the display (490) of FIG. 3), at least one processor including processing circuitry (e.g., the processor (120, 230) of FIGS. 1, 2, and / or 4), and a memory (e.g., the memory (130, 240) of FIGS. 1, 2, and / or 4). In one embodiment, the memory may store instructions that, when individually and / or collectively executed by the at least one processor, cause the electronic device (101) to perform operations.

[0149] In one embodiment, the instructions, when individually and / or collectively executed by at least one processor, may cause the electronic device to execute an interactive service. The instructions, when individually and / or collectively executed by at least one processor, may cause the electronic device to receive natural language input from a user while executing the interactive service. The instructions, when individually and / or collectively executed by at least one processor, may cause the electronic device to generate a text prompt including the natural language, based on the natural language input, to generate a plurality of tasks for the natural language. The instructions, when individually and / or collectively executed by at least one processor, may cause the electronic device to provide the text prompt to an on-device and / or server-generated artificial intelligence (AI). The instructions, when individually and / or collectively executed by at least one processor, may cause the electronic device to obtain data comprising a plurality of tasks based on the text prompt. The instructions, when individually and / or collectively executed by at least one processor, may cause the electronic device to classify the plurality of tasks based on the data according to performing entities of at least two of a user of the electronic device, an external device, and / or an application of the electronic device, and to assign at least one task to each performing entity. The instructions, when individually and / or collectively executed by at least one processor, may cause the electronic device to configure at least one guide and / or at least one menu for executing the at least one task assigned to each performing entity.The above instructions, when individually and / or collectively executed by at least one processor, may cause the electronic device to display on the display the at least one guide and / or the at least one menu, dividing the at least two execution entities.

[0150] According to one embodiment, the instructions, when individually and / or collectively executed by at least one processor, may cause the electronic device to determine information related to a conversation topic, a task, a user of the electronic device, an external device, and / or an application of the electronic device for the natural language, and to generate the text prompt based on the natural language and the information.

[0151] According to one embodiment, the instructions, when individually and / or collectively executed by at least one processor, may cause the electronic device to obtain, from the generative AI based on the text prompt, the data including a title related to a conversation topic, the plurality of tasks, and information about an executing entity related to the plurality of tasks.

[0152] According to one embodiment, the instructions, when individually and / or collectively executed by at least one processor, may cause the electronic device to identify a performing entity and a task based on the data, classify the tasks according to the performing entity, assign at least one task to each performing entity, identify at least one guide and / or at least one menu for executing a defined action according to the assigned task from the performing entity, and provide an interface in which the at least one guide and / or the at least one menu are connected in a hierarchical structure.

[0153] According to one embodiment, the at least one menu may include the capabilities of the performing entity performing the task and parameters for controlling actions based on the capabilities. According to one embodiment, the capabilities may include functions provided for the task. According to one embodiment, the parameters may include setting values ​​related to the functions.

[0154] According to one embodiment, the instructions, when individually and / or collectively executed by at least one processor, may cause the electronic device to identify the capability and the parameter based on the data, set the parameter for operating the capability and a function according to the capability, and configure at least one menu including the capability and / or the parameter.

[0155] According to one embodiment, the instructions, when individually and / or collectively executed by at least one processor, may cause the electronic device to identify a title related to a conversation topic based on the data, generate a title menu including the title, and provide the title menu and the at least one menu and / or the at least one guide as an interface connected in a hierarchical structure.

[0156] According to one embodiment, the instructions, when individually and / or collectively executed by at least one processor, cause the electronic device to display the at least one menu and / or the at least one guide based on a time-series characteristic.

[0157] According to one embodiment, the at least one guide and / or the at least one menu may include task control information related to the at least two performing entities performing the plurality of tasks.

[0158] According to one embodiment, the instructions, when individually and / or collectively executed by at least one processor, may cause the electronic device to receive an input for selecting a portion corresponding to the at least one menu, determine, in response to the input, an executing entity related to the selected menu and action information to be executed by the executing entity, and execute an action related to an application, a predetermined function of the application, and / or a predetermined function of a linked device based on the executing entity and the action information.

[0159] According to one embodiment, the at least one menu may include spatial information and route guide information regarding the execution of the execution entity. According to one embodiment, the spatial information and route guide information may be displayed by being arranged on a spatial diagram provided by the electronic device.

[0160] According to one embodiment, the instructions, when individually and / or collectively executed by at least one processor, may cause the electronic device to display the spatial information and the route guide information by arranging them on a spatial diagram provided by the electronic device.

[0161] According to one embodiment, the instructions, when individually and / or collectively executed by at least one processor, may cause the electronic device to obtain the spatial diagram defined in relation to the performing subject and to display information related to the activity path of the performing subject based on the spatial diagram.

[0162] Hereinafter, an operating method of an electronic device (e.g., the electronic device (101, 201) of FIGS. 1, 2, and / or 4) (hereinafter, the electronic device (101)) according to various embodiments will be described in detail. Operations performed in the electronic device (101) according to various embodiments may be executed by a processor (e.g., the processor (120, 230) of FIGS. 1, 2, and / or 4) including various processing circuitry and / or executable program elements of the electronic device (101). According to one embodiment, the operations performed in the electronic device (101) may be stored as instructions in a memory (130) and individually and / or collectively performed (or executed) by the processor (120, 220).

[0163] FIG. 5 is a flowchart illustrating a method of operating an electronic device according to one embodiment of the present disclosure.

[0164] According to one embodiment, FIG. 5 may illustrate an example of a method for supporting an interactive service in an electronic device (101) according to one embodiment. For example, the electronic device (101) may illustrate an example of a method for generating a prompt based on natural language, obtaining result data (e.g., a response text for a user's query (or inquiry)) based on the generated prompt, and providing output data (e.g., including a guide and a menu) generated (or processed) based on the result data.

[0165] A method for supporting an interactive service in an electronic device (101) according to one embodiment of the present disclosure may be performed, for example, according to a flowchart illustrated in FIG. 5. The flowchart illustrated in FIG. 5 is an example according to one embodiment of an operation of the electronic device (101), and the order of at least some operations may be changed or performed in parallel, performed as independent operations, or at least some other operations may be performed complementarily to at least some operations. According to one embodiment of the present disclosure, operations 501 to 517 may be performed in at least one processor of the electronic device (101) (e.g., processors 120 and 230 of FIGS. 1, 2, and / or 4).

[0166] As illustrated in FIG. 5, an operation method performed by an electronic device (101) according to an embodiment may include an operation of executing an interactive service (operation 501), an operation of receiving a natural language input (operation 503), an operation of generating a prompt based on the natural language (operation 505), an operation of obtaining result data related to the prompt (operation 507), an operation of classifying a user's task-specific guide based on the result data (operation 509), an operation of identifying an action corresponding to each of the classified guides (operation 511), an operation of configuring at least one menu for executing an action (operation 513), an operation of generating output data based on the guide and the menu (operation 515), and an operation of displaying output data including the guide and the menu in a time series manner (operation 517).

[0167] Referring to FIG. 5, in operation 501, the processor (120) of the electronic device (101) may execute an interactive service. According to one embodiment, the processor (120) may display an execution screen of an application (e.g., an interactive service application, an assistant application, or an AI agent) on the display. According to one embodiment, the processor (120) may receive an input related to the execution of the application from a user. According to one embodiment, the processor (120) may execute the application and display the execution screen of the application on the display in response to the input related to the execution of the application.

[0168] In operation 503, the processor (120) may receive a natural language input. According to one embodiment, the processor (120) may receive a text-based natural language input entered through a keypad on the execution screen and / or a voice-based natural language input entered through a microphone.

[0169] In operation 505, a prompt may be generated based on natural language. According to one embodiment, the processor (120) may generate a prompt (or instruction) to provide a response (e.g., a guide for a task) corresponding to the natural language based on the natural language input. According to one embodiment, the processor (120) may generate a prompt to answer a user's query based on the natural language input. According to one embodiment, the processor (120) may analyze a context (or user intention) based on the natural language, determine a context related to a task to be performed by the user based on the context (e.g., perform situational awareness), determine reference information (e.g., user information, contact information, device information, application information) based on the context, and perform operations related to generating a prompt based on the natural language and the reference information.

[0170] In one embodiment, the processor (120) may provide a prompt to the generative AI to provide result data (e.g., a response to a user query) based on the prompt. In one embodiment, the prompt may be provided to the on-device generative AI and / or to the server-side generative AI (e.g., an LLM server). In one embodiment, the result data may be provided based on the on-device and / or server.

[0171] In operation 507, the processor (120) may obtain result data in relation to the prompt (or instruction). In one embodiment, the processor (120) may obtain (or generate) result data according to an answer generation process (e.g., an answer generation operation based on natural language) executed in relation to the prompt (or instruction) in the on-device artificial intelligence. In one embodiment, the processor (120) may obtain (or receive) result data according to an answer generation process (e.g., an answer generation operation based on natural language) executed in relation to the prompt (or instruction) in the server artificial intelligence from the server.

[0172] In operation 509, the processor (120) may classify the user's task-specific guide based on the result data. In one embodiment, the processor (120) may extract tasks based on the result data and classify the guide (e.g., conversation) based on the task.

[0173] In operation 511, the processor (120) can identify actions corresponding to each classified guide. In one embodiment, the processor (120) can determine an agent to perform a task according to the guide, and determine capabilities and parameters for the agent. In one embodiment, the processor (120) can determine a defined action according to the task based on the capabilities and parameters.

[0174] In operation 513, the processor (120) may configure at least one menu for executing an action. In one embodiment, the processor (120) may determine a menu that causes the performing entity to execute a defined action according to a task. In one embodiment, the menu may include parameters for controlling the capabilities of the performing entity for performing the task and the actions according to the capabilities. In one embodiment, the capabilities may include functions (or services or operations) provided for the task. In one embodiment, the parameters may include setting values ​​related to the functions.

[0175] In operation 515, the processor (120) may generate output data based on the guide and menu. According to one embodiment, the processor (120) may generate and provide an interface that comprehensively provides task control information and each performing entity that performs multiple tasks based on the result data.

[0176] In operation 517, the processor (120) may display output data including guides and menus in a time-series manner. According to one embodiment, the processor (120) may control the display to display output data generated (or reconstructed) based on result data acquired in relation to a prompt (or instruction). According to one embodiment, the processor (120) may display output data (e.g., guides and menus) generated based on the result data on an execution screen (e.g., a conversation screen of an interactive service) via the display.

[0177] FIGS. 6, 7 and 8 are drawings illustrating an example of an interface supporting an interactive service in an electronic device according to one embodiment and an operation example of providing output data using the interface.

[0178] According to one embodiment, FIGS. 6, 7 and 8 illustrate an example of a conversation screen (or conversation interface) of an interactive service, and may be an example diagram for comparing a conversation screen of an interactive service according to the present disclosure with a general conversation screen.

[0179] According to one embodiment, FIG. 6 may illustrate an example of a generally provided dialogue screen. According to one embodiment, FIG. 7 may illustrate an example of a dialogue screen (e.g., a dialogue screen of a first display type) provided according to one embodiment of the present disclosure. According to one embodiment, FIG. 8 may illustrate an example of a dialogue screen (e.g., a dialogue screen of a second display type) provided according to one embodiment of the present disclosure.

[0180] As illustrated in Figure 6, conventionally, text-based result data (e.g., answers) may be provided in a simple listing in response to a user query (or inquiry). For example, answers may consist of complex information, such as tasks that the user must perform directly, operating a device, or using an application. However, conventionally, due to the simple text-based listing, if the user wishes to perform an action by referring to the answer, they may have to individually perform each required action, which can be complex.

[0181] As illustrated in FIGS. 7 and 8 , an electronic device (101) according to one embodiment may provide a menu (or interface or UX component) that can individually and / or collectively execute tasks (or actions) (e.g., user tasks, device tasks, and / or application tasks) related to text (e.g., answer content) in response to a user query (or inquiry). For example, as illustrated in FIGS. 7 and 8 , result data (e.g., answers) may be classified into task-specific guides, and at least one menu for task (or action) execution may be mapped for each guide, thereby providing regenerated result data (e.g., output data).

[0182] In one embodiment, FIG. 7 may illustrate an example of a conversation screen provided as a first display type in an electronic device (101). In one embodiment, FIG. 8 may illustrate an example of a conversation screen provided as a second display type in an electronic device (101).

[0183] In one embodiment, a conversation screen of the first display type may be an example of providing response content and menus in a chronological order, grouped and separated by task. For example, the first display type may be a tile that displays conversation sentences and menus together. In one embodiment, a conversation screen of the second display type may be an example of providing menus in a chronological order, grouping them and distinguishing them from response content (e.g., displaying them using a higher layer). For example, the second display type may be a type that displays conversation sentences and menus separately.

[0184] According to one embodiment, as in the examples of FIGS. 7 and / or 8, the electronic device (101) can analyze a conversation context in a conversation sentence. The electronic device (101) can extract capabilities and parameters associated with the execution of device operations, services, and / or application functions within the conversation sentence, and display a menu that enables immediate execution of the corresponding operation based at least on the capabilities and parameters. In one embodiment, the menu can include parameters for controlling actions according to the capabilities and capabilities of the performing subject (or executing subject) performing (or executing) the task. In one embodiment, the capabilities can include functions (or services or operations) provided for the task, as in the examples of below. In one embodiment, the parameters can include setting values ​​related to the functions, as in the examples of below.

[0185] Definition Example Capability: The function (or action) performed by a performing entity (e.g., device, application) and / or the service provided or the scope of work that can be performed by the performing entity (1) Washing function of a washing machine (e.g., washing, rinsing, spinning) (2) Refrigeration function of a refrigerator (e.g., rapid cooling, automatic temperature control) (3) Order tracking function of a food delivery application (e.g., checking real-time order status, providing expected delivery time) Parameter: A set value or variable that affects the action performed by a performing entity (e.g., device, application) (used to adjust or set a specific function or action) (1) Washing temperature of a washing machine (e.g., about 30 degrees, about 40 degrees, about 60 degrees) (2) Cooking time in an oven (e.g., about 30 minutes, about 45 minutes, about 1 hour) (3) Notification frequency in an application (e.g., immediately, once a day, at a defined time)

[0186] In one embodiment, a user can use a menu to directly perform a task (or action). For example, the menu may execute a corresponding function based on user input. In one embodiment, the menu, when selected by the user, may provide a function to navigate to a corresponding page displaying application movement and capabilities. In one embodiment, the menu, when selected by the user, may provide a function to automatically input and execute device capabilities and parameters. In one embodiment, the menu, when selected by the user, may provide a function to automatically input and execute application capabilities and parameters for tasks that would otherwise be performed directly by a human (e.g., the user). In one embodiment, capabilities and / or parameters may be manually selected by the user and then input as changed values, and may be automatically changed based on additional conversation context.

[0187] According to one embodiment, the electronic device (101) can extract a title based on a conversation context (e.g., Dinner with Sally) and provide the extracted title using a card (e.g., a title card) at the top of the conversation screen.

[0188] According to one embodiment, FIGS. 7 and 8 may illustrate examples of operations for supporting an interactive service based on artificial intelligence in an electronic device (101) according to one embodiment. According to one embodiment, the artificial intelligence may include generative AI. Generative AI may refer to an AI technology that newly creates similar content using existing content such as text, audio, and / or images. For example, generative AI may refer to an AI technology that can generate content (e.g., text, audio, image, and / or video) corresponding to an input based on a given input. According to one embodiment, the electronic device (101) may generate (e.g., regenerate or reconstruct) and provide data based on generative AI (e.g., on-device AI). According to one embodiment, the electronic device (101) may request data generation from a server, and receive and provide data generated based on the generative AI of the server from the server. According to one embodiment, the electronic device (101) may provide a prompt (or instruction or generative AI prompt) to the generative AI requesting data generation (e.g., a question or instruction to be entered into the generative AI).

[0189] In one embodiment, data (e.g., result data and / or output data) may be generated based on text-based and / or speech-based natural language input by a user. In one embodiment, natural language input by a user may be used as a prompt source for data generation.

[0190] According to one embodiment, as illustrated in FIG. 7 or FIG. 8, the method of operation performed by the electronic device (101) according to one embodiment may include an operation of receiving a prompt source (e.g., natural language) related to data generation based on interaction with a user, and generating (e.g., regenerating or reconstructing) and providing data on a server or on-device based on the prompt source.

[0191] FIG. 9 is a diagram illustrating an example of a menu provided by an electronic device according to one embodiment of the present disclosure.

[0192] According to one embodiment, FIG. 9 may illustrate an example of providing multiple menus as a group (or bundled object).

[0193] According to one embodiment, when multiple devices (e.g., an oven and an induction) need to be operated (or performed) simultaneously, the electronic device (101) may provide related menus together as a group (or bundled object). According to one embodiment, the electronic device (101) may provide a shortcut object (or button) that can batch-execute (or simultaneously execute or control) multiple tasks based on the menu.

[0194] According to one embodiment, the electronic device (101) may provide a relevant object (or icon) for user authentication (e.g., fingerprint recognition, PIN number, pattern) through a corresponding menu for a task requiring user confirmation, such as payment.

[0195] FIGS. 10, 11, and 12 are drawings illustrating various examples of menus provided by an electronic device according to one embodiment of the present disclosure.

[0196] According to one embodiment, FIGS. 10, 11, and 12 may illustrate examples of generating and providing a menu with device information (e.g., speaker, light, TV), content (e.g., playlist), option information (e.g., volume, brightness, channel), spatial information (e.g., Living room), and / or additional information (e.g., image, map) that a user must operate based on result data (or answer content).

[0197] For example, an example may be presented where a user inputs a natural language (e.g., a user query) such as "How should I set the mood for a home party?" and receives an answer (or result data) regarding a task related to three categories of music, lighting, and display in response to the natural language. For example, upon receiving the result data, the electronic device (101) may identify the three categories of music, lighting, and display, extract capabilities and / or parameters according to the guide of each category, and provide a menu for each guide.

[0198] According to one embodiment, as in the example of FIG. 10, the electronic device (101) may extract device information (e.g., speaker), content (e.g., playlist) and option information (e.g., volume) that the user should operate based on the result data (e.g., first guide part about music), and provide a first menu that can control the device (e.g., speaker).

[0199] According to one embodiment, as in the example of FIG. 11, the electronic device (101) may extract device information (e.g., lighting), space information (e.g., Living room) and option information (e.g., brightness) that the user must operate based on the result data (e.g., the second guide part regarding lighting), and provide a second menu that can control the corresponding device (e.g., lighting).

[0200] According to one embodiment, as in the example of FIG. 12, the electronic device (101) may extract device information (e.g., TV), content, and additional information (e.g., TV image) that the user should operate based on the result data (e.g., the third guide part regarding the display), and provide a third menu that can control the corresponding device (e.g., TV). According to one embodiment, in a conversation situation where the generative artificial intelligence suggests recommended content to be executed on a device (e.g., an external device) such as a TV and / or a speaker, the electronic device (101) may provide content assets (asserts) (e.g., video content or music content) that the user has, or, if there is no content asset, may provide new content by generating it based on the generative artificial intelligence.

[0201] According to one embodiment, the electronic device (101) may obtain result data corresponding to natural language, analyze the result data, and generate output data including guides and menus based on the result data, as shown in the examples of FIGS. 10 to 12, and provide the output data to the user.

[0202] FIGS. 13 and 14 are drawings illustrating examples of providing a menu in an electronic device according to one embodiment of the present disclosure.

[0203] According to one embodiment, FIGS. 13 and 14 may illustrate an example of providing a menu related to controlling a device (e.g., a robot vacuum cleaner) that a user must operate based on result data (or response content).

[0204] For example, a user may input natural language (e.g., a user query) such as, "I'm inviting Sally over for a house party this weekend. What should I do?" and, in response to the natural language, a result data (or response) such as, "How about starting with cleaning? It can efficiently clean dusty areas."

[0205] According to one embodiment, as in the example of FIG. 13 or FIG. 14, the electronic device (101) may analyze (or extract) capabilities (e.g., 'cleaning function' of the robot vacuum cleaner) and parameters (e.g., operation timer value) based on the result data, and provide a menu related to control of the robot vacuum cleaner based on the capabilities and parameters.

[0206] According to one embodiment, as illustrated in FIG. 14, the electronic device (101) may provide a spatial representation related to an executing subject based on a menu. For example, the electronic device (101) may switch to and display the menu of the example of FIG. 14 based on the selection of the menu in the example of FIG. 13, or may directly display the menu of the example of 14, thereby providing a spatial representation (or visual information) about the movement line (or range) of the activity (or operation) of the user / device. For example, the electronic device (101) may determine a cleaning space (e.g., spatial information or geographical information) based on defined map information of the robot cleaner, and may spatially express the cleaning space of the robot cleaner by displaying visual information about the activity movement line of the robot cleaner through a spatial drawing (1400) (e.g., map view) through a menu.

[0207] According to one embodiment, the electronic device (101) may obtain a spatial diagram (1400) (e.g., call a defined map or create a map) by matching the spatial device list and control information of other electronic devices (e.g., Internet of Things devices) connected to the electronic device (101) in the vicinity, and may provide a spatial representation, such as a movement line of the device (e.g., movement guide information), based on the map. According to one embodiment, when the electronic device (101) determines that a dynamic task is performed that involves movement within the space based on the display of a menu, the electronic device (101) may provide a user with a visual display of the movement path using a two-dimensional (2D) or three-dimensional (3D) map view.

[0208] FIG. 15 is a diagram illustrating an example of an operation for supporting an interactive service in an electronic device according to one embodiment of the present disclosure.

[0209] According to one embodiment, FIG. 15 illustrates an example of displaying dialogue sentences and menus separately. For example, FIG. 15 illustrates an example of displaying menus corresponding to each task separately, displaying the separated menus together with dialogue sentences, but providing them independently of the dialogue sentences.

[0210] Referring to FIG. 15, a user may use an electronic device (101) to execute a conversational service based on an application (e.g., a conversational service application, an assistant application, or an AI agent), and input natural language corresponding to the user's query (or inquiry) on the execution screen (e.g., a conversation screen) of the application. For example, while executing the conversational service, the user may input, "I'm inviting Sally over for a house party this weekend. What should I prepare? I'd like steak with cabbage garnish. Tell me the recipe and how to cook it." In one embodiment, the natural language input may be input as a voice-based input based on the user's speech, and / or as a text-based input based on a keypad. According to one embodiment, in response to receiving the user's natural language input, the electronic device (101) may display text (e.g., a conversation sentence or conversation content) corresponding to the input natural language.

[0211] According to one embodiment, the electronic device (101) can analyze (or determine) the conversation context based on text corresponding to natural language. According to one embodiment, the electronic device (101) can extract a title (e.g., Dinner with Sally) based on the conversation context. According to one embodiment, the electronic device (101) can provide the extracted title using a card (e.g., a title card) at the top of the conversation screen. According to one embodiment, when the electronic device (101) receives a natural language input, it can generate a prompt (or instruction) that generates result data (e.g., an answer) based on the natural language. According to one embodiment, the electronic device (101) can provide the prompt to a generative AI to provide an answer based on the prompt. According to one embodiment, the prompt can be provided to an on-device generative AI and / or a server-side generative AI. In one embodiment, answers to user queries may be provided on-device and / or server-based.

[0212] According to one embodiment, the electronic device (101) may determine a conversation topic, a task, an action performer for each task, a device related to the action performer, and / or an application related to the action performer based on a natural language input, and may generate a prompt based on the natural language and the determination result. For example, the electronic device (101) may use the determination result as input for a prompt source.

[0213] According to one embodiment, the electronic device (101) may display the acquired result data while executing an interactive service. According to one embodiment, the electronic device (101) may acquire the result data in relation to a prompt (e.g., natural language). According to one embodiment, the electronic device (101) may acquire (or generate) the result data according to an answer generation process (e.g., an answer generation operation based on natural language) executed in relation to a prompt (or instruction) in on-device artificial intelligence. According to one embodiment, the electronic device (101) may acquire (or receive) the result data according to an answer generation process (e.g., an answer generation operation based on natural language) executed in relation to a prompt (or instruction) in server artificial intelligence from a server. For example, the electronic device (101) may acquire first result data such as the example of below while executing an interactive service.

[0214] Here's the recipe for a steak and cabbage garnish. Ingredients: Beef steak meat, half a head of cabbage, 2 tablespoons butter, 2 tablespoons olive oil, a pinch of salt, a pinch of pepper. The ingredients I don't have in the fridge right now but can order right now are steak and cabbage.

[0215] According to one embodiment, the electronic device (101) may display the first result data on a dialogue screen (or display). According to one embodiment, the electronic device (101) may control the display to display the first result data acquired in relation to a prompt (or instruction). According to one embodiment, the result data may be provided as N or more result data (e.g., multiple result data). For example, the electronic device (101) may acquire multiple result data based on generative artificial intelligence and provide the multiple result data. According to one embodiment, the electronic device (101) may analyze (or determine) the first result data. According to one embodiment, the electronic device (101) may analyze the context (or sentence elements) based on the first result data and provide a menu for a task (or action) based on the result of the analysis. According to one embodiment, the electronic device (101) may provide a menu for an order task. For example, the electronic device (101) may display a menu (e.g., an order menu) containing information (or query summary information) about the user's query (e.g., order -> curry) based on the first result data. In one embodiment, the order menu providing information about the user's query may be displayed for a certain period of time. For example, the order menu may be displayed until a menu related to action execution (e.g., an action menu) is displayed, and if the action menu is generated, the order menu may not be displayed. Without limitation, the order menu may be continuously displayed while the interactive service is being executed.

[0216] In one embodiment, a user may input natural language related to confirmation or execution of the first result data. For example, a user may input "I want to order" while executing an interactive service. In one embodiment, the electronic device (101) may, in response to receiving the user's natural language input, display text (e.g., a conversation sentence or conversation content) corresponding to the input natural language. In one embodiment, when the electronic device (101) receives the natural language input, it may generate a prompt (or instruction) that generates result data (e.g., an answer) based on the natural language. In one embodiment, the electronic device (101) may provide a prompt to the generative artificial intelligence to provide an answer based on the prompt.

[0217] According to one embodiment, the electronic device (101) may obtain second result data such as the example in below in relation to the prompt.

[0218] Here's how to cook it: First, preheat the oven to 450℉ (about 230℃). Wash the cabbage and slice it thinly.

[0219] According to one embodiment, the electronic device (101) may display the second result data on a dialogue screen (or display). According to one embodiment, the electronic device (101) may control the display to display the second result data obtained in relation to the prompt (or instruction). According to one embodiment, the electronic device (101) may analyze (or determine) the second result data. According to one embodiment, the electronic device (101) may analyze the context (or sentence elements) based on the second result data and extract capabilities (e.g., oven) and parameters (e.g., about 230°C). According to one embodiment, the electronic device (101) may provide a menu for a task (or action) based on the capabilities and parameters. For example, the electronic device (101) may display a menu (e.g., an oven control menu) including capabilities and parameters for the task.

[0220] In one embodiment, the electronic device (101) may analyze the context (or sentence elements) based on the second result data and extract capabilities (e.g., "Reminder") and parameters (e.g., "Cabbage Slicing"). In one embodiment, the electronic device (101) may display a menu (e.g., "Reminder" menu) for tasks (or actions) based on the capabilities and parameters. For example, the electronic device (101) may display capabilities and parameters of an application (e.g., "Reminder" application) for a task that the user must perform.

[0221] FIG. 16 is a diagram illustrating an example of an operation for supporting an interactive service in an electronic device according to one embodiment of the present disclosure.

[0222] According to one embodiment, FIG. 16 illustrates an example of displaying dialogue sentences and menus separately. For example, FIG. 15 illustrates an example of displaying menus corresponding to each task separately, and displaying these menus alongside dialogue sentences, but providing them independently of the dialogue sentences. According to one embodiment, FIG. 16 illustrates an example of creating a routine based on a menu generated upon completion of a dialogue on a defined topic.

[0223] Referring to FIG. 16, a user can use an interactive service by performing actions corresponding to those described in the description section referring to FIG. 15. According to one embodiment, while executing an interactive service, the electronic device (101) may receive a natural language input from the user, obtain result data related to the user's task in relation to the natural language, and generate and provide at least one menu based on the result data.

[0224] In one embodiment, as illustrated in FIG. 16, a user can complete a conversation on a topic conducted through an interactive service. For example, the user can input defined natural language associated with the completion of the conversation (e.g., "thank you," "end," "complete," "end"), and the generative AI can recognize the defined natural language, recognize the completion of the conversation, and provide result data corresponding to the completion of the conversation.

[0225] According to one embodiment, when the electronic device (101) determines that a conversation on a given topic has been completed, it may generate a routine based on a menu generated based on the topic. According to one embodiment, the electronic device (101) may generate routine data based on a combination of menus related to the topic. In one embodiment, the routine data may include at least one configuration information (or parameter or data) related to the conditions and actions that constitute the routine. For example, the routine data may include the capabilities and parameters of a menu related to executing a task performed by the user.

[0226] In one embodiment, the electronic device (101) may guide the creation of a routine based on a title menu (or title card) related to the subject matter. For example, the electronic device (101) may provide an indicator (or object) that directs the execution of a routine or routine in a defined portion of the title menu. In one embodiment, the electronic device (101) may change the title menu to a routine menu when creating a routine.

[0227] FIG. 17 is a diagram illustrating an example of an operation for supporting an interactive service in an electronic device according to one embodiment of the present disclosure.

[0228] According to one embodiment, FIG. 17 may illustrate an example of executing a routine generated based on completion of a conversation or editing routine data.

[0229] Referring to FIG. 17, when a conversation is completed, the electronic device (101) can provide a routine menu by combining menus based on tasks (or actions) that must be performed directly by the device, application, and user, related to menus generated (or tasks executed) according to the conversation flow.

[0230] According to one embodiment, a user may perform an input to select a routine menu. According to one embodiment, when the electronic device (101) receives an input through the routine menu, it may display a combination of action menus constituting the routine menu in a time-series configuration in the form of a list.

[0231] In one embodiment, the combination of action menus for a routine can be changed (or edited) by the user. In one embodiment, the user can change the order of the menus using a defined input (e.g., a drag-and-drop touch gesture). In one embodiment, the user can remove menus of unwanted actions (or steps) from the routine using a defined input (e.g., a flick touch gesture or a swipe touch gesture). For example, the electronic device (101) can change the order of the routine and / or remove at least one menu based on the user's input, and create (or update) the routine with the changed combination of menus.

[0232] According to one embodiment, the electronic device (101) can execute a routine based on a defined input from a user for executing the routine (e.g., a tap touch gesture for selecting an indicator). For example, the electronic device (101) can execute the routine by executing the corresponding task based on the order in which the tasks are performed and defined conditions (e.g., conditions such as execution conditions, such as execution time zone and / or execution location).

[0233] FIGS. 18, 19, and 20 are diagrams illustrating examples of operations for executing a routine in an electronic device according to one embodiment of the present disclosure.

[0234] According to one embodiment, FIGS. 18, 19 and 20 may illustrate examples in which routines are executed and displayed in chronological order using previously generated menu combinations.

[0235] As illustrated in FIG. 18, the electronic device (101) can execute a routine by automatically executing tasks (or actions) of a menu based on time sequence when defined conditions (e.g., defined date and time) related to the execution of the routine are met.

[0236] According to one embodiment, FIGS. 18 to 20 may illustrate examples of routines generated from combinations of menus created under the theme of “Dinner with Sally.”

[0237] In one embodiment, as illustrated in FIG. 18, the electronic device (101) may display menus in a time sequence in which combinations of previously generated menus should be executed when a condition for executing a routine is met (e.g., on the day of dinner with Sally).

[0238] In one embodiment, as illustrated in FIG. 18, the menu may be provided such that first information (e.g., icons and names) related to a performing entity (e.g., a device, a service, an application, or a user (or a user experience)) is arranged in a left column, and second information related to capabilities and / or parameters is arranged in a right column. In one embodiment, the first information may include information related to the performing entity, and the second information may be provided as an expression of an experience unit that includes the performing entity. For example, in FIG. 18, when provided as an expression of an experience unit that includes multiple performing entities, such as "Set a mood," the second information may summarize and display icons, capabilities, and parameters for each detailed topic.

[0239] In one embodiment, the electronic device (101) can determine whether the tasks in each menu are performable based on capabilities and parameters based on the menu. For example, the electronic device (101) can determine whether an error condition exists, such as an abnormality (or error) detected in a device and / or application that must (or is scheduled to) operate.

[0240] In one embodiment, as illustrated in FIGS. 19 and 20 , when the electronic device (101) determines an error situation, it may determine another device and / or another application that can replace the device and / or application in which the error occurred. In one embodiment, the electronic device (101) may create (or update) a routine by changing the task to be performed based on the other device and / or other application. For example, the electronic device (101) may re-create a list of menus with replaceable options depending on the context.

[0241] According to one embodiment, as illustrated in FIGS. 19 and 20 , when a machine for making a sauce suitable for food is broken, the electronic device (101) may change the task to ordering a commercially available sauce and provide it. For example, when the electronic device (101) determines an error situation for the "Sauce maker" as illustrated in FIG. 19 , the electronic device (101) may provide a notification of the error situation in the first information and / or second information portion. For example, as illustrated in FIG. 20 , the electronic device (101) may change the menu for performing a task by the "sauce maker" to a task for ordering a sauce, such as "Order Sauce."

[0242] FIGS. 21A, 21B, 21C, 21D, and 21E are diagrams illustrating examples of operations for supporting interactive services in an electronic device according to one embodiment of the present disclosure.

[0243] According to one embodiment, FIGS. 21a, 21b, 21c, 21d, and 21e may illustrate examples of generating (or regenerating) and providing result data and menus when the situational context related to a conversation topic changes.

[0244] As illustrated in FIG. 21a, a user may input natural language (e.g., "What should I eat for dinner tonight?") related to the user's query while a dialogue screen is displayed following the execution of the conversational service. According to one embodiment, the electronic device (101) may generate a prompt (or instruction) based on the natural language input. According to one embodiment, the electronic device (101) may provide a prompt to the generative artificial intelligence to provide a response based on the prompt.

[0245] As illustrated in FIG. 21b, the electronic device (101) can obtain result data related to the prompt and display the result data on the dialogue screen. For example, the electronic device (101) can display the result data for "Hello Rose, Jaden won't be home until tomorrow, so I'll prepare a meal for two to share with my husband."

[0246] According to one embodiment, the electronic device (101) can analyze (or determine) the result data. According to one embodiment, the electronic device (101) can analyze the context (or sentence elements) based on the result data, and provide a menu for a task (or action) based on the result of the analysis. For example, the electronic device (101) can generate and provide an executable menu corresponding to the task “2-person meal ingredients -> Add to B Mart shopping cart.” According to one embodiment, the electronic device (101) can extract a list of applications that can match the result data (e.g., natural language or answers) and the capabilities of the applications (e.g., functions and / or services) and set them as parameters (or data values) of the menu.

[0247] As illustrated in FIG. 21c, the electronic device (101) can detect changes in the conversation context while executing an interactive service. For example, the electronic device (101) can detect changes in the conversation context based on task execution conditions, such as detecting the voice of a third party (e.g., a son) through the microphone of the electronic device (101), detecting the location of a third party in the vicinity based on situational awareness, or determining the expected visit of a third party based on a received message, while executing an interactive service. For example, the electronic device (101) can detect the voice of a third party (e.g., a son, Jaden, who is scheduled to be absent) (e.g., "Mom, I'm home~") through the microphone.

[0248] In one embodiment, when the electronic device (101) detects a change in context, it may generate a prompt (or instruction) based on previously input natural language information related to the change (e.g., son Jaden's attendance). In one embodiment, the electronic device (101) may provide a prompt to the generative artificial intelligence to provide a response based on the prompt.

[0249] As illustrated in FIG. 21d, the electronic device (101) can obtain result data related to the prompt and display the result data on the dialogue screen. For example, the electronic device (101) can display the result data "Jaden is about to arrive home. I will change the meal plan to a three-person meal."

[0250] According to one embodiment, the electronic device (101) can analyze (or determine) the result data. According to one embodiment, the electronic device (101) can analyze the context (or sentence elements) based on the result data, and provide a menu for a task (or action) based on the result of the analysis. For example, the electronic device (101) can generate and provide an executable menu corresponding to a task corresponding to "3-person meal ingredients -> Add to B Mart shopping cart." For example, the electronic device (101) can provide the menu by changing the parameters. According to one embodiment, the electronic device (101) can extract a list of applications that can match the result data (e.g., natural language or answers) and the capabilities of the applications (e.g., functions and / or services) and set them as parameters (or data values) of the menu.

[0251] According to one embodiment, FIG. 21E may illustrate an example of additionally providing a menu (or indicator or object) for a user to change a task and / or for a user's confirmation in relation to a change in result data. As illustrated in FIG. 21E , the electronic device (101) may provide multiple menus (e.g., a first option menu, a second option menu, and a third option menu) on the dialogue screen for changing options or for a user's confirmation in response to changed result data (e.g., "Jaden is about to arrive home, so I'll change the meal to a 3-person meal"). For example, the first option menu may be "Change to meal delivery," the second option menu may be "Contact Jaden," and the third option menu may be "Order groceries as is." Options that can be changed and / or added and / or menus for a user's confirmation may be provided in response to the result data.

[0252] According to one embodiment, the electronic device (101) may generate (e.g., regenerate) and provide a menu with applied performers, capabilities, and parameters, taking into account the changed options, based on additional input from the user (e.g., selection of an options menu).

[0253] FIG. 22 is a flowchart illustrating a method of operating an electronic device according to one embodiment.

[0254] According to one embodiment, FIG. 22 may illustrate an example of a method for supporting an interactive service in an electronic device (101) according to one embodiment. For example, the electronic device (101) may illustrate an example of a method for generating a prompt (e.g., a text prompt) based on natural language, obtaining data (e.g., a response text for a user's query (or inquiry)) based on the generated prompt, and providing data (e.g., including a guide and a menu) generated (or processed) based on the data.

[0255] A method for supporting an interactive service in an electronic device (101) according to one embodiment of the present disclosure may be performed, for example, according to a flowchart illustrated in FIG. 22. The flowchart illustrated in FIG. 22 is an example according to one embodiment of an operation of the electronic device (101), and the order of at least some operations may be changed or performed in parallel, performed as independent operations, or at least some other operations may be performed complementarily to at least some operations. According to one embodiment of the present disclosure, operations 2201 to 2217 may be performed in at least one processor of the electronic device (101) (e.g., processors 120 and 230 of FIGS. 1, 2, and / or 4).

[0256] According to one embodiment, the operations described in FIG. 22 may be heuristically performed in combination with the operations described in FIGS. 5 to 22e, for example, or heuristically performed as a replacement for at least some of the operations described and combined with at least some other operations, or heuristically performed as a detailed operation of at least some of the operations described.

[0257] As illustrated in FIG. 22, an operation method performed by an electronic device (101) according to an embodiment includes an operation of executing an interactive service (operation 2201), an operation of receiving an input of natural language (operation 2203), an operation of generating a text prompt including natural language to generate a plurality of tasks for the natural language (operation 2205), an operation of providing the text prompt to a generative AI of an on-device and / or a server (operation 2207), an operation of obtaining data including a plurality of tasks based on the text prompt (operation 2209), an operation of classifying a plurality of tasks based on the data according to at least two performing entities of a user of the electronic device, an external device (e.g., a device), and / or an application of the electronic device (operation 2211), an operation of assigning at least one task to each performing entity (operation 2213), an operation of configuring at least one guide and / or at least one menu for executing at least one task assigned to each performing entity (operation 2215), and displaying at least one guide and / or at least one menu on a display. May include actions (action 2217).

[0258] Referring to FIG. 22, in operation 2201, the processor (120) of the electronic device (101) may execute an interactive service. According to one embodiment, the processor (120) may display an execution screen of an application (e.g., an interactive service application, an assistant application, or an AI agent) on the display. According to one embodiment, the processor (120) may receive an input related to the execution of the application from a user. According to one embodiment, the processor (120) may execute the application and display the execution screen of the application on the display in response to the input related to the execution of the application.

[0259] In operation 2203, the processor (120) may receive a natural language input. According to one embodiment, the processor (120) may receive a natural language input from a user while executing an interactive service. According to one embodiment, the processor (120) may receive a text-based natural language input entered through a keypad on an execution screen and / or a voice-based natural language input entered through a microphone.

[0260] In operation 2205, the processor (120) may generate a text prompt including natural language to generate a plurality of tasks for the natural language. According to one embodiment, the processor (120) may generate a text prompt (or instruction) to provide a response corresponding to the natural language (e.g., data for a task for each performing entity) based on the input of the natural language. According to one embodiment, the processor (120) may determine information related to a conversation topic, a task, a user of the electronic device (101), an external device (e.g., a device), and / or an application of the electronic device (101) for the natural language, and generate a text prompt based on the natural language and the information.

[0261] According to one embodiment, the processor (120) may generate a text prompt based on natural language input to answer a user's query based on the natural language. According to one embodiment, the processor (120) may perform operations related to analyzing a context (or user intent) based on the natural language, determining a context related to a task to be performed by the user based on the context (e.g., performing situational awareness), determining reference information based on the context (e.g., user information, contact information, device information, application information), and generating a text prompt based on the natural language and the reference information.

[0262] In operation 2207, the processor (120) may provide a text prompt to a generative AI of the on-device and / or server. In one embodiment, the processor (120) may provide (or transmit) the text prompt to the generative AI to provide data (e.g., a response to a user query) based on the text prompt. In one embodiment, the text prompt may be provided to the generative AI of the on-device and / or the generative AI of the server (e.g., an LLM server). In one embodiment, the data may be provided based on the on-device and / or the server.

[0263] In operation 2209, the processor (120) may obtain data including a plurality of tasks based on a text prompt. In one embodiment, the processor (120) may obtain data including a title related to a conversation topic, a plurality of tasks, and information about an executor related to the plurality of tasks from a generative AI based on the text prompt.

[0264] In one embodiment, the processor (120) may obtain (or generate) data according to an answer generation process (e.g., an answer generation operation based on natural language) executed in relation to a text prompt in the on-device AI. In one embodiment, the processor (120) may obtain (or receive) data from the server according to an answer generation process (e.g., an answer generation operation based on natural language) executed in relation to a prompt in the server AI.

[0265] In operation 2211, the processor (120) may classify a plurality of tasks based on data according to at least two performing entities among a user of the electronic device, an external device (e.g., a device), and / or an application of the electronic device. According to one embodiment, the processor (120) may extract (or identify) performing entities and tasks from the data, and classify tasks according to each performing entity.

[0266] In operation 2213, the processor (120) may assign at least one task to each performing entity. According to one embodiment, the processor (1200) may classify tasks by performing entity and assign at least one task to each performing entity.

[0267] In operation 2215, the processor (120) may configure at least one guide and / or at least one menu for executing at least one task assigned to each performing entity. According to one embodiment, the processor (120) may determine (or identify) at least one guide and / or at least one menu that allows the performing entity to execute a defined action according to the assigned task. According to one embodiment, the processor (120) may identify capabilities and parameters based on data, set parameters for operating functions according to the capabilities and capabilities, and configure at least one menu including the capabilities and / or parameters. In one embodiment, the guide and / or menu may include capabilities of the performing entity performing the task and parameters for controlling actions (or operations) according to the capabilities. In one embodiment, the capabilities may include functions (or services or operations) provided for the tasks. In one embodiment, the parameters may include setting values ​​related to the functions.

[0268] According to one embodiment, at least one guide and / or at least one menu may include task control information related to at least two performing entities performing multiple tasks. According to one embodiment, at least one menu may include spatial information and movement guide information on which performing entities are executed. According to one embodiment, the spatial information and movement guide information may be displayed by being arranged on a spatial diagram (e.g., the spatial diagram (1400) of FIG. 14) provided by the electronic device (101). According to one embodiment, the processor (120) may obtain a spatial diagram defined in relation to the performing entity, and display information related to the activity movement line of the performing entity based on the spatial diagram.

[0269] In operation 2217, the processor (120) may display at least one guide and / or at least one menu on the display. According to one embodiment, the processor (120) may provide an interface in which at least one guide and / or at least one menu are connected in a hierarchical structure. According to one embodiment, the processor (120) may divide at least one guide and / or at least one menu into at least two execution entities and display them on the display in a hierarchical structure. According to one embodiment, the processor (120) may display data (e.g., output data) including the guide and the menu based on time-series characteristics. According to one embodiment, the processor (120) may display data (e.g., the guide and the menu) generated based on the data on an execution screen (e.g., a conversation screen of an interactive service) through the display.

[0270] According to one embodiment, the processor (120) may receive an input for selecting a portion corresponding to at least one menu displayed while displaying at least one guide and / or at least one menu through the display. According to one embodiment, the processor (120) may, in response to the input for selecting a portion corresponding to the menu, determine an execution entity related to the selected menu and action information to be executed by the execution entity. According to one embodiment, the processor (120) may execute an action related to an application, a predetermined function of the application, and / or a predetermined function of a linked device based on the execution entity and the action information.

[0271] An operating method performed in an electronic device (101) according to one embodiment of the present disclosure may include an operation of executing an interactive service. The operating method may include an operation of receiving a natural language input from a user while executing the interactive service. The operating method may include an operation of generating a text prompt including the natural language based on the natural language input, so as to generate a plurality of tasks for the natural language. The operating method may include an operation of providing the text prompt to a generative artificial intelligence (AI) of an on-device and / or a server. The operating method may include an operation of acquiring data including a plurality of tasks based on the text prompt. The operating method may include an operation of classifying the plurality of tasks based on the data according to at least two performing entities among a user of the electronic device, an external device, and / or an application of the electronic device, and allocating at least one task to each performing entity. The operating method may include an operation of configuring at least one guide and / or at least one menu for executing the at least one task assigned to each performing entity. The above method of operation may include an operation of displaying on a display at least one guide and / or at least one menu by dividing them into at least two performing entities.

[0272] According to one embodiment, the act of generating the text prompt may include the act of determining information related to a conversation topic, a task, a user of the electronic device, an external device, and / or an application of the electronic device for the natural language, and the act of generating the text prompt based on the natural language and the information.

[0273] In one embodiment, the act of obtaining the data may include an act of obtaining, from the generative AI, the data including a title related to a conversation topic, the plurality of tasks, and information about an executor related to the plurality of tasks, based on the text prompt.

[0274] According to one embodiment, the operating method may include an operation of identifying a performing subject and a task based on the data, an operation of classifying the tasks according to the performing subject, an operation of assigning at least one task to each performing subject, an operation of identifying at least one guide and / or at least one menu that causes the performing subject to execute a defined action according to the assigned task, and an operation of providing an interface in which the at least one guide and / or the at least one menu are connected in a hierarchical structure.

[0275] According to one embodiment, the at least one menu may include the capabilities of the performing entity performing the task and parameters for controlling actions based on the capabilities. According to one embodiment, the capabilities may include functions provided for the task. According to one embodiment, the parameters may include setting values ​​related to the functions.

[0276] According to one embodiment, the operating method may include an operation of identifying the capability and the parameter based on the data, an operation of setting the parameter for operating the capability and a function according to the capability, and an operation of configuring at least one menu including the capability and / or the parameter.

[0277] According to one embodiment, the operating method may include an operation of identifying a title related to a conversation topic based on the data, an operation of generating a title menu including the title, and an operation of providing the title menu and the at least one menu and / or the at least one guide as an interface connected in a hierarchical structure.

[0278] According to one embodiment, the at least one menu and / or the at least one guide may be displayed based on a time-series characteristic.

[0279] According to one embodiment, the at least one menu may include spatial information and route guide information on which the performing subject is executed.

[0280] According to one embodiment, the spatial information and the route guide information may be displayed by being placed on a spatial drawing provided by the electronic device.

[0281] A non-transitory computer-readable recording medium storing instructions that, when executed by a processor (120) of an electronic device (101) according to one embodiment of the present disclosure, cause the processor (120) to perform operations, wherein the instructions, when executed by the processor, cause the electronic device to perform an interactive service, an operation of receiving a natural language input from a user while executing the interactive service, an operation of generating a text prompt including the natural language to generate a plurality of tasks for the natural language based on the input of the natural language, an operation of providing the text prompt to a generative artificial intelligence (AI) of an on-device and / or a server, an operation of obtaining data including a plurality of tasks based on the text prompt, an operation of classifying the plurality of tasks based on the data according to at least two performing entities of a user of the electronic device, an external device, and / or an application of the electronic device, and allocating at least one task to each performing entity, and each The recording medium may include an operation of configuring at least one guide and / or at least one menu for executing at least one task assigned to each performing subject, and an operation of displaying the at least one guide and / or the at least one menu on a display by dividing the at least two performing subjects.

[0282] It will be appreciated that the above-described embodiments and their technical features may be combined with each other in any and all combinations, as long as there is no potential conflict between the two embodiments or features. For example, any and all combinations of two or more of the above-described embodiments may be envisioned and incorporated within the present disclosure. One or more features from any embodiment may be incorporated into any other embodiment, providing a corresponding advantage or advantages.

[0283] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

[0284] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another component (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0285] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0286] Various embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more commands stored in a storage medium (or recording medium) (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one command among the one or more commands stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one command called. The one or more commands may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g. electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0287] According to one embodiment, the method according to various embodiments disclosed in this document may be provided as a computer program product. The computer program product may be traded between sellers and buyers as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or may be provided through an application store (e.g., Play Store). TM ) or directly between two user devices (e.g., smart phones), online distribution (e.g., downloading or uploading). In the case of online distribution, at least a part of the computer program product may be at least temporarily stored or temporarily created in a machine-readable storage medium (or recording medium), such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0288] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and arranged in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0289] The various embodiments of the present disclosure disclosed in this specification and drawings are intended to provide specific examples to facilitate easy explanation of the technical content of the present disclosure and to aid understanding of the present disclosure, and are not intended to limit the scope of the present disclosure. Therefore, the scope of the present disclosure should be interpreted to include all modifications or variations derived based on the technical concepts of the present disclosure, in addition to the embodiments disclosed herein.

Claims

1. In the electronic device (101, 201), display; At least one processor (120, 230) comprising processing circuitry; and Includes a memory (130, 240) for storing instructions, The above instructions, when executed by the at least one processor, cause the electronic device to: Run an interactive service, While executing the above interactive service, natural language input is received from the user, Based on the input of the natural language, a text prompt including the natural language is generated to generate a plurality of tasks for the natural language, Provide the above text prompt to the on-device and / or server-generated artificial intelligence, Obtaining data including multiple tasks based on the above text prompt, Based on the above data, the plurality of tasks are classified according to the performing subjects of at least two of the user of the electronic device, the external device, and / or the application of the electronic device, and at least one task is assigned to each performing subject. Configure at least one guide and / or at least one menu for the execution of at least one task assigned to each of the above-mentioned execution subjects, and An electronic device that displays at least one guide and / or at least one menu on the display by dividing the at least two performing entities.

2. In the first paragraph, when the instructions are executed by the at least one processor, the electronic device, Determine information related to the conversation topic, task, user of the electronic device, external device, and / or application of the electronic device in the natural language; An electronic device that generates the text prompt based on the natural language and the information.

3. In the first paragraph, when the instructions are executed by the at least one processor, the electronic device, An electronic device that obtains data from the generative AI based on the text prompt, the data including a title related to the conversation topic, the plurality of tasks, and information about the performer related to the plurality of tasks.

4. In the first paragraph, when the instructions are executed by the at least one processor, the electronic device, Identify the performer and task based on the above data, Classify the above tasks according to the above performing subject, Assign at least one task to each of the above performing entities, Identify at least one guide and / or at least one menu that causes the execution of a defined action according to the task assigned by the above performing entity, An electronic device providing an interface in which at least one guide and / or at least one menu are connected in a hierarchical structure.

5. In paragraph 1, At least one of the above menus includes a capability of the performing subject performing the task and a parameter for controlling the action according to the capability, The above capabilities include functions provided for tasks, An electronic device wherein the above parameters include setting values ​​related to the above function.

6. In the fifth paragraph, when the instructions are executed by the at least one processor, the electronic device, Based on the above data, identify the above capabilities and the above parameters, Set the above parameters that operate the above capabilities and functions according to the above capabilities, An electronic device configured to configure at least one menu including the above capabilities and / or the above parameters.

7. In the sixth paragraph, when the instructions are executed by the at least one processor, the electronic device, Based on the above data, identify titles related to conversation topics, Create a title menu containing the above title, Provide an interface in which the above title menu and the at least one menu and / or the at least one guide are connected in a hierarchical structure, An electronic device wherein at least one menu and / or at least one guide are displayed based on time-series characteristics.

8. In paragraph 1, An electronic device wherein said at least one guide and / or said at least one menu includes task control information related to said at least two performing entities performing said plurality of tasks.

9. In the first paragraph, when the instructions are executed by the at least one processor, the electronic device, Receiving an input for selecting a portion corresponding to at least one of the above menus, In response to the above input, determine the execution subject related to the selected menu and the action information to be executed by the execution subject, An electronic device that executes an action related to an application, a defined function of the application, and / or a defined function of a linked device based on the above-mentioned performing subject and the above-mentioned action information.

10. In the first paragraph, at least one menu includes spatial information and route guide information on which the performing subject is executed, An electronic device in which the above spatial information and the above route guide information are displayed by being placed on a spatial drawing provided by the electronic device.

11. In the first paragraph, when the instructions are executed by the at least one processor, the electronic device, Obtain the above spatial drawing defined in relation to the above performing subject, An electronic device that displays information related to the activity path of the performing subject based on the above spatial drawing.

12. In the operating method of an electronic device (101, 201), The action of running an interactive service; An action of receiving natural language input from a user while executing the above interactive service; An operation of generating a text prompt including the natural language, based on the input of the natural language, to generate a plurality of tasks for the natural language; An act of providing the above text prompt to an on-device and / or server-generated artificial intelligence (AI); An operation of obtaining data including a plurality of tasks based on the above text prompt; An operation of classifying the plurality of tasks based on the data and assigning at least one task to each performing entity, such as a user of the electronic device, an external device, and / or an application of the electronic device; An operation of configuring at least one guide and / or at least one menu for the execution of at least one task assigned to each of the above-mentioned execution subjects; and A method comprising an action of displaying on a display at least one guide and / or at least one menu by dividing the at least two performing entities.

13. In the 12th paragraph, the action of generating the text prompt is: An action to determine information related to a conversation topic, task, user of an electronic device, external device, and / or application of the electronic device in relation to the above natural language; A method comprising an action of generating the text prompt based on the natural language and the information.

14. In the 12th paragraph, the operation of acquiring the data is as follows: A method comprising an action of obtaining data from the generative AI based on the text prompt, the data including a title related to the conversation topic, the plurality of tasks, and information about an executor related to the plurality of tasks.

15. In paragraph 12, An action to identify the performer and task based on the above data; An action to classify the above tasks according to the above performing subject, An action of assigning at least one task to each of the above performing subjects, An action identifying at least one guide and / or at least one menu that causes the execution of a defined action according to a task assigned by the above performing entity; A method comprising providing an interface in which at least one guide and / or at least one menu are connected in a hierarchical structure.

Images