Electronic device and method for supporting action execution

Abstract

An electronic device and a method for supporting action execution are disclosed. The electronic device can: acquire content indicating a task to be performed, and acquire, on the basis of the content, a description of at least one type of work required to perform the task; and extract device information of the electronic device, generate, on the basis of the description and the device information, a program for a set of actions to be executed in the electronic device, and display a user interface related to the program. Other various embodiments identified through the present document are possible.

Description

Electronic devices and methods supporting action execution

[0001] The present disclosure relates to an electronic device and a method of operating the same, and more particularly, to an electronic device and method supporting action execution.

[0002] Electronic devices are increasingly equipped with complex functions, such as taking photos and videos, playing music and videos, playing games, receiving broadcasts, and supporting wireless Internet, and are being implemented as comprehensive multimedia devices. Accordingly, electronic devices are evolving into new forms, both hardware and software, to satisfy user needs while enhancing portability and convenience.

[0003] Additionally, as artificial intelligence (AI) technology advances and generative AI models emerge, various attempts are being made to provide services for new content expanded from user input, rather than being limited to that input.

[0004] The above information may be provided as background information to aid in understanding the present disclosure. None of the above is claimed to be prior art related to the present disclosure or can be used to determine prior art related to the present disclosure.

[0005] An electronic device according to one embodiment of the present disclosure may include a display, at least one processor, and a memory storing instructions. The instructions, when executed by the at least one processor, may cause the electronic device to obtain content indicating a task to be performed, obtain a description of at least one work required to perform the task based on the content, extract device information of the electronic device, generate a program for a set of actions to be executed in the electronic device based on the description and the device information, and display a user interface related to the program through the display.

[0006] A method of operating an electronic device according to one embodiment of the present disclosure may include an operation of obtaining content indicating a task to be performed, an operation of obtaining a description of at least one work required to perform the task based on the content, an operation of extracting device information of the electronic device, an operation of generating a program for a set of actions to be executed in the electronic device based on the description and the device information, and an operation of displaying a user interface related to the program through a display of the electronic device.

[0007] A storage medium according to one embodiment of the present disclosure may be a non-transitory computer-readable storage medium. The storage medium may record a program for executing a method of operating an electronic device. The storage medium may record a program for executing a method including an operation of obtaining guide content indicating a task to be performed, an operation of obtaining an operation sequence description for operations required to perform the task based on the content, an operation of extracting device information of the electronic device, an operation of generating a program for a set of actions to be performed in the electronic device based on the description and the device information, and an operation of displaying a user interface related to the program through a display of the electronic device.

[0008] FIG. 1 is a block diagram of an electronic device within a network environment according to various embodiments.

[0009] FIG. 2 is a block diagram of an electronic device according to one embodiment.

[0010] FIG. 3 is a flowchart illustrating a method by which an electronic device supports action execution according to one embodiment.

[0011] FIG. 4 is an example of a user interface associated with a process of generating a program using a video by an electronic device according to one embodiment.

[0012] FIG. 5 is an example of a user interface associated with a process in which an electronic device displays a user interface associated with a program according to one embodiment.

[0013] FIG. 6 is a diagram illustrating a process in which an electronic device generates a program from a video using at least one generative AI model according to one embodiment.

[0014] FIG. 7 is an example of a user interface associated with a process for editing a description of an electronic device according to one embodiment.

[0015] FIG. 8 is an example of a user interface related to a process of sharing a program by an electronic device according to one embodiment.

[0016] FIG. 9 is an example of a user interface associated with a process of generating a program using text by an electronic device according to one embodiment.

[0017] FIG. 10 is a diagram illustrating a process in which an electronic device generates a program from text using at least one generative AI model according to one embodiment.

[0018] FIG. 11 is an example of a user interface associated with a process for setting a user routine using a simple action function in an electronic device according to one embodiment.

[0019] FIG. 12 is an example of a user interface associated with a process for setting user routines and routine execution conditions using a simple action function in an electronic device according to one embodiment.

[0020] FIG. 13 is an example of a user interface related to a process in which an electronic device obtains content for program creation using external content according to one embodiment.

[0021] FIG. 14 is an example of a process in which an electronic device according to one embodiment provides a simple action function using an intelligent assistant.

[0022] FIG. 15 is another example of a process in which an electronic device according to one embodiment provides a simple action function using an intelligent assistant.

[0023] FIG. 16 is another example of a process in which an electronic device according to one embodiment provides a simple action function using an intelligent assistant.

[0024] FIG. 17 is another example of a process in which an electronic device according to one embodiment provides a simple action function using an intelligent assistant.

[0025] FIG. 18 is a diagram illustrating a system including a generative AI model according to one embodiment.

[0026] The hardware specifications and software execution environments of electronic devices may vary from device to device, and the range of user interfaces may also vary.

[0027] Because functions or services that can be performed with user commands (e.g., voice commands) depend on the individual device characteristics (e.g., manufacturer, hardware specifications, OS version) of each electronic device, support for new services or features across diverse interfaces may be limited. For example, while electronic devices can relatively easily perform simple functions based on direct user commands, such as "turn up the volume" or "set the alarm," complex scenarios requiring multiple actions across different applications may be difficult to implement.

[0028] Furthermore, because various services are implemented through complex user interfaces, even if documentation or videos are provided to guide users, it can be difficult for them to easily understand the operations required for the desired service. For example, while macro applications can be created and used for repetitive tasks, even using macro applications can present limitations in reflecting diverse requirements due to individual device characteristics or changes in the execution environment.

[0029] According to various embodiments of the present disclosure, the user experience can be improved by providing a personalized interface for a task to be performed according to the individual device characteristics or execution environment of the electronic device.

[0030] According to various embodiments of the present disclosure, multiple operations related to a task can be performed using a simple and intuitive interface, without complex and difficult navigation procedures. This reduces human error during the multiple operations and enables free sharing of the interface among multiple users.

[0031] According to various embodiments of the present disclosure, functions requiring the performance of different operations between heterogeneous applications can be seamlessly implemented through a unified interface while reducing user inconvenience.

[0032] The effects that can be obtained from the present disclosure are not limited to the effects mentioned above, and other effects that are not mentioned can be clearly understood by a person having ordinary skill in the art to which the present disclosure belongs from the description below.

[0033] Hereinafter, the present disclosure relates to an electronic device and method that support action execution.

[0034] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. In connection with the description of the drawings, the same or similar reference numerals may be used for identical or similar components. Furthermore, in the drawings and related descriptions, descriptions of well-known functions and configurations may be omitted for clarity and conciseness.

[0035] FIG. 1 is a block diagram of an electronic device (101) within a network environment (100) according to various embodiments. Referring to FIG. 1, in the network environment (100), the electronic device (101) may communicate with the electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with the electronic device (104) or a server (108) via a second network (199) (e.g., a long-range wireless communication network). In one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0036] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or operations. According to one embodiment, as at least a part of the data processing or operations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or an auxiliary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together with the main processor (121). For example, when the electronic device (101) includes the main processor (121) and the auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a given function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0037] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, in the electronic device (101) itself where artificial intelligence is performed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0038] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0039] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0040] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0041] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0042] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0043] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0044] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0045] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0046] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0047] The haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. According to one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0048] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0049] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0050] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0051] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). The corresponding communication module within these communication modules can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can use subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196) to verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199).

[0052] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0053] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas, for example, by the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device via the at least one selected antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0054] According to various embodiments, the antenna module (197) may form a mmWave antenna module. In one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

[0055] At least some of the above components may be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI)).

[0056] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0057] In one embodiment, a "work" may be a task and / or a component of content indicating the task. For example, a task may be a unit of work (or an individual task) required to perform the task. For example, a task may include at least one task (or information about at least one task). For example, at least one task required to perform the task or included in the task may be identified (or extracted) from content indicating the task.

[0058] In one embodiment, an "action" may be a component of a program. For example, the action may be referred to as an "operation" or a "function." For example, the action may be a unit action (or individual action) set by the program or included in the program. For example, the program may be configured to set and / or execute (e.g., support execution, automatic execution, or simple execution) at least one action. For example, at least one action may be identified (or extracted) from at least one of the content or a description generated based on the content.

[0059] In one embodiment, a "program" may be for setting and / or executing an action set (or at least one action). For example, a program may include an action set (or at least one action).

[0060] According to one embodiment, the program may include (or provide) UI information (e.g., object information, UI element information, image information, action-specific UI screen information).

[0061] In one embodiment, the work of a task may correspond to an action of a program.

[0062] In one embodiment, the program may include at least one action corresponding to at least some of the multiple tasks included in the task. For example, the program may include at least one action corresponding to all or some of the multiple tasks included in the task.

[0063] According to one embodiment, a program (or a set of actions set by the program) may include at least one action. For example, the action may include at least one of a first action, a second action, a third action, a fourth action, or a fifth action. The first action may be identical to or similar to the task. The second action may be a task processed and / or modified based on at least a portion of device information of an electronic device (e.g., the electronic device (200) of FIG. 2 ). The third action may be a task divided into parts. The fourth action may be a combination of two or more tasks. The fifth action may be a newly added action related to the task. For example, the action may include at least one of an action executed through interaction with a user of the electronic device (e.g., the electronic device (200) of FIG. 2)) (or an action requiring user input), or an action executed without user interaction (or an action that does not require user input). For example, the action may include at least one of an action that executes at least one of an application, a function that can be provided by the application, or a function that can be provided by an electronic device (e.g., the electronic device (200) of FIG. 2). For example, the action may include at least one of an action that requires a UI display, or an action that does not require a UI display. For example, the action may include at least one of an action that requires linkage with an external electronic device (e.g., the electronic device (102, 104) of FIG. 1, the server (108) of FIG. 1), an action that is executed through the external electronic device, or an action that requires installation of a new application. For example, the action may include an action that automates interaction with a UI (e.g., automatic input such as touch, scroll, drag, zoom-in, zoom-out, motion, or volume control).

[0064] FIG. 2 is a block diagram of an electronic device (200) according to one embodiment.

[0065] Referring to FIG. 2, an electronic device (200) according to one embodiment may include a processor (210), a memory (220), and a display (230). The electronic device (200) may further include a communication circuit (240). The electronic device (200) of FIG. 2 may correspond to the electronic device (101) illustrated in FIG. 1. The processor (210), the memory (220), the display (230), and the communication circuit (240) included in the electronic device (200) of FIG. 2 may correspond to the processor (120), the memory (130), the display module (160), and the communication module (190) of FIG. 1, respectively.

[0066] In some embodiments, at least one of the illustrated components of the electronic device (200) may be omitted, integrated with one another, or additionally provided with other components. The processor (210), memory (220), display (230), and communication circuit (240) included in the electronic device (200) may be electrically and / or operatively connected to one another to exchange signals (e.g., commands or data) therebetween.

[0067] According to one embodiment, the processor (210) (e.g., the processor (120) of FIG. 1) may include at least one processor. The processor (210) may execute various functions supported by the electronic device (200). The processor (210) may control the memory (220), the display (230), and / or the communication circuit (240). The processor (210) may execute an application and control various hardware by executing code written in a programming language stored in the memory (220) of the electronic device (200).

[0068] According to one embodiment, the memory (220) (e.g., the memory (130) of FIG. 1) may store instructions that, when executed by the processor (210), cause the electronic device (200) to perform various operations. For example, the processor (120) of the electronic device (200) may execute instructions stored in the memory (220) to provide an easy action function.

[0069] According to one embodiment, the display (230) (e.g., the display module (160) of FIG. 1) may include at least one of a liquid crystal display (LCD), a thin film transistor LCD (TFT-LCD), organic light emitting diodes (OLED), a light emitting diode (LED), an active matrix organic LED (AMOLED), a flexible display, and a 3-dimensional display. In some embodiments, some of these displays may be configured as transparent or light-transmitting so that the outside may be viewed therethrough. For example, the transparent or light-transmitting display may be configured as a transparent display including a transparent OLED (TOLED).

[0070] According to one embodiment, the display (230) may be implemented as an integral part with a touch panel. The display (230) may support a touch function and may detect a user input (e.g., a touch) and transmit it to the processor (210). A display (230) supporting a touch function may be referred to as a touch screen. The display (230) may further include a structure capable of detecting an input using a stylus pen, such as an electro-magnetic resonance (EMR) or an active electrostatic solution (AES).

[0071] In one embodiment, the communication circuit (240) (e.g., the communication module (190) of FIG. 1) may establish a communication connection with one or more external electronic devices (e.g., the electronic devices (102, 104) of FIG. 1, or the server (108) of FIG. 1) and transmit and receive various data. For example, the communication circuit (240) may support at least one communication method among cellular communication, the Internet, Wi-Fi, Bluetooth, near field communication (NFC), or ultra-wide band (UWB) communication.

[0072] According to one embodiment, an electronic device (200) (e.g., the electronic device (101) of FIG. 1) may provide a function that supports action execution (hereinafter referred to as a “simple action function”). In one embodiment, the electronic device (200) may provide the simple action function using at least one generative artificial intelligence (AI) model.

[0073] According to one embodiment, the processor (210) of the electronic device (200) can manage applications and service packages (or capsules, domains) installed in the electronic device (200) and a designated external electronic device (e.g., an Internet of Things device) (e.g., the electronic device (102) of FIG. 1). The processor (210) can manage functions provided by each application (e.g., deeplink, shortcut, intent), and can update the corresponding application when each application is installed / updated / deleted.

[0074] According to one embodiment, the processor (210) of the electronic device (200) may obtain content indicating a task to be performed. According to one embodiment, the content may include various types of content (e.g., text, voice, audio, image (still image), video, multimedia). For example, the content may be content including information about at least one task required to perform the task.

[0075] In one embodiment, the content representing the task to be performed may be selected based on user input of the electronic device (200).

[0076] According to one embodiment, the content representing the task to be performed may be generated by an external electronic device (e.g., electronic device (102, 104) of FIG. 1) that is an electronic device of another user and transmitted to the electronic device (200).

[0077] In one embodiment, the content representing the task to be performed may be received through a conversation session (e.g., a conversation session for a chat app) in which at least one message is exchanged between a user of the electronic device (200) and another user of an external electronic device (e.g., the electronic devices (102, 104) of FIG. 1 ). For example, the conversation session may correspond to a communication channel between the electronic device (200) and the external electronic device. The conversation session may be established (or configured) through the communication circuit (240).

[0078] In one embodiment, the content representing the task to be performed may be received by an intelligent assistant interacting with the user of the electronic device (200) via voice.

[0079] According to one embodiment, the content representing the task to be performed may be generated through handwriting input. For example, the electronic device (200) may receive text corresponding to the handwriting input through the display (230). The text may be written by the user using a stylus pen (or electronic pen). The text may be text related to the task to be performed (e.g., create a morning routine) or text related to at least one task (e.g., play XX music in the morning). The electronic device (200) may identify the task or at least one task related to the text using a generative AI model. The operation of identifying the task or at least one task from the handwriting input may be initiated (or triggered) based on a designated input. For example, the designated input may be an input of drawing a stroke of a designated shape around the handwriting input.

[0080] According to one embodiment, the processor (210) may obtain a description (or specification) of at least one operation required to perform the task based on content representing the task to be performed. For example, the description may correspond to a description (or explanation) of the work procedures required to perform the task.

[0081] In one embodiment, the description may include data regarding at least some of the multiple operations necessary for performing the task or constituting the task. For example, the description may include text describing at least one unit operation (or individual operation) necessary for performing the task. For example, the description may include text describing at least some of the unit operations necessary for performing the task, regardless of the order. For example, the description may include text describing at least some of the unit operations having continuity in a stepwise or sequential manner.

[0082] According to one embodiment, the processor (210) can obtain a description generated based on the content using at least one generative AI model.

[0083] According to one embodiment, the processor (210) can obtain a description generated based on the content by inputting the content representing the task into at least one generative AI model.

[0084] According to one embodiment, the processor (210) can obtain a description generated based on content using the first AI model and the second AI model.

[0085] For example, the processor (210) can obtain (e.g., extract) an image data set generated by the first AI model by inputting a video corresponding to the content (e.g., a guide video) into the first AI model (e.g., the frame analysis AI model (611) of FIG. 6). The processor (210) can obtain the description generated by the second AI model by inputting the image data set into the second AI model (e.g., the LVM model (612) and / or the LLM model (613) of FIG. 6).

[0086] For example, image embedding data (e.g., 622 in FIG. 6) extracted from a video and text embedding data (e.g., 623 in FIG. 6) extracted through the LVM model (612) can be used as inputs to the LLM model (613) to obtain a description. The electronic device (200) can obtain a description by inputting the image embedding data and the text embedding data into the LLM model (613).

[0087] For example, the processor (210) can extract a text data set by inputting text corresponding to content (e.g., guide text) into a first AI model (e.g., the action parsing AI model (1011) of FIG. 10). The processor (210) can obtain the description generated by the second AI model by inputting the text data set into a second AI model (e.g., the LLM model (1013) of FIG. 10).

[0088] In one embodiment, the first AI model and the second AI model used to obtain the description may be the same AI model. The electronic device (200) can obtain a description generated based on content using the same AI model.

[0089] According to one embodiment, the processor (210) can extract device information of the electronic device (200). For example, the processor (210) can extract device information from the memory (220).

[0090] According to one embodiment, the processor (210) can manage applications and / or service packages (or capsules, domains) installed in the electronic device (200) and a designated external electronic device (e.g., the electronic device (101, 104) of FIG. 1, the server (108)). The processor (210) can manage functions (or actions, e.g., deeplink, shortcut, intent) provided by each application.

[0091] According to one embodiment, the processor (210) may update the functions (or actions) according to the installation / update / delete operation of the application. The processor (210) may store function-related information (or action-related information, e.g., definitions, usage instructions, and user interface information for each function or action) that may be provided through at least one application installed on the electronic device (200) in the memory (220), and may manage the function-related information.

[0092] According to one embodiment, the device information of the electronic device (200) may include at least some of application information (e.g., information about the manufacturer, version, category, type, application identifier, application UI, application function, etc.) for application(s) installed in the electronic device (200), software information (e.g., information about the operating system (OS) version, user interface (UI) / user experience (UX) version, etc.), or hardware information (e.g., information about the device type, device identifier, specifications, camera, biometric sensor, etc.). The device information may include, for example, status information of the electronic device (200) (e.g., remaining battery, whether charging, type of connected network, whether the screen is locked, location).Device information includes, for example, information about at least one application running in the foreground (e.g., a name of at least one application running in the foreground, a category to which at least one application running in the foreground belongs, and / or a description of at least one application running in the foreground), screen context information (e.g., information about UI objects included in a screen currently being displayed through the electronic device (200), information about text included in a screen currently being displayed through the electronic device (200), information about an image included in a screen currently being displayed through the electronic device (200), information about a function included in a screen currently being displayed through the electronic device (200), activity information performed during a specified period of time (e.g., during the past month, after the electronic device (200) was switched from an off state to an on state) or after the electronic device (200) was connected to a specific network (e.g., information about applications that were run, information about the time at which a specified application was run, information about a location at which a specified application was run, and / or information about text input by a user), and / or electronic It may include information on input / output of at least one application installed in the device (200). The processor (210) may also obtain (e.g., receive) device information of an external electronic device (e.g., an electronic device (102, 104) of FIG. 1) (e.g., an Internet of Things device connected to a user's home network, a wearable electronic device registered to the user's user account), for example.

[0093] According to one embodiment, the processor (210) may generate a program (e.g., an action executable file) for a set of actions (at least one action) to be executed on the electronic device (200) based on a description of at least one operation required to perform a task and device information of the electronic device (200). The set of actions may include at least one action.

[0094] In one embodiment, depending on the device information, different programs or different sets of actions may be generated.

[0095] According to one embodiment, at least one action included in the above program or at least one action set included in the above program may be selected based on device information. For example, a function that can be executed in an application running in the foreground may be determined as at least one action. For example, a function indicated by text information displayed on the screen may be determined as at least one action. For example, a function included in an application that has been used most frequently during a specified period may be determined as at least one action. For example, a function included in an application that can provide an output corresponding to the result of performing a task may be determined as at least one action. For example, different programs may be generated based on the remaining battery of the electronic device (200), whether it is charging, the type of network connected, whether the screen is locked, and / or the location of the electronic device (200).

[0096] According to one embodiment, the processor (210) may generate a program (e.g., an action execution file) for a set of actions (at least one action) to be executed in the electronic device (200) based further on device information of the external electronic device (e.g., an Internet of Things device, a wearable electronic device of a user).

[0097] According to one embodiment, the action set may include at least one action (or function) executable by the electronic device (200).

[0098] According to one embodiment, the action set may include at least one action (or function) that the electronic device (200) can execute in conjunction with a designated external electronic device (e.g., electronic device (102, 104) of FIG. 1) (e.g., an Internet of Things device connected to the user's home network, a wearable electronic device registered to the user's user account).

[0099] According to one embodiment, the action set may include a series of action steps configured by a generative AI model that are not provided by the manufacturer in the default setting state of the electronic device (200).

[0100] According to one embodiment, the action set may include an action for installing an application. For example, if a first application capable of executing an action required for performing a task is not installed on the electronic device (200), an action for installing the first application or a second application corresponding to the first application may be included in the action set. For example, the processor (210) may display a user interface element guiding the installation of the first application or the second application through the display (230). The processor (210) may install the first application or the second application on the electronic device (200) in response to a user input (e.g., an installation approval input) for the interface element.

[0101] According to one embodiment, the processor (210) can generate a program for the action set by inputting a description of at least one task and device information of the electronic device (200) into a third artificial intelligence model (e.g., the MLLM model (614) of FIG. 6 or the MLLM model (1014) of FIG. 10).

[0102] According to one embodiment, the processor (210) can obtain a description generated based on the content by inputting the content representing the task into at least one generative AI model.

[0103] According to one embodiment, the processor (210) can generate a description and / or program based on content using at least one generative AI model.

[0104] According to one embodiment, the electronic device (200) can obtain a description generated based on content using a first AI model (e.g., at least one of the frame analysis AI model (611) of FIG. 6 or the action parsing AI model (1011) of FIG. 10) and / or a second AI model (e.g., at least one of the LVM model (612) of FIG. 6, the LLM model (613) or the LLM model (1013) of FIG. 10). The electronic device (200) can obtain a program generated based on the content using a third AI model (e.g., the MLLM model (614) of FIG. 6 or the MLLM model (1014) of FIG. 10).

[0105] In one embodiment, at least some of the first AI model, the second AI model, or the third AI model may be the same AI model (e.g., a unified AI model).

[0106] According to one embodiment, at least some of the first AI model, the second AI model, and the third AI model may be provided within the electronic device (200) in the form of on-device AI models.

[0107] According to one embodiment, at least some of the first AI model, the second AI model, or the third AI model may be provided from outside the electronic device (200) (e.g., the server (108) of FIG. 1).

[0108] In one embodiment, some of the first AI model, the second AI model, and the third AI model may be provided internally in the form of on-device AI models of the electronic device (200). Others of the first AI model, the second AI model, and the third AI model may be provided externally of the electronic device (200) (e.g., the server (108) of FIG. 1).

[0109] In some embodiments, additional information about the video (411) (e.g., link information, download path information, related website information) may be additionally input to the at least one generative AI model for program generation. In such a case, the at least one generative AI model may provide a program generated based on at least a portion of the video (411), the device information, and the additional information about the video (411).

[0110] According to one embodiment, the program may be for setting and / or executing a set of actions (at least one action) to be executed on the electronic device (200).

[0111] In one embodiment, the program may include a set of actions. The program may be for setting and / or executing a set of actions related to a task (e.g., execution assistance, automatic execution, simple execution).

[0112] In one embodiment, the program may include UI information. The program may be configured to provide a user interface associated with at least one action.

[0113] According to one embodiment, the processor (210) can obtain a program generated (e.g., programmed) by the third AI model by inputting at least a portion of a description for at least one task and device information of the electronic device (200) into the third AI model. For example, the electronic device (200) can generate a program for an action set by inputting at least a portion of the description for at least one task and device information of the electronic device (200) into the third AI model (e.g., the MLLM model (614) of FIG. 6 or the MLLM model (1014) of FIG. 10).

[0114] According to one embodiment, the program may be configured to set and / or execute a set of actions (at least one action) to be executed on the electronic device (200). For example, when the program is executed, a plurality of actions set by the program may be automatically executed in batches. The program may group or package into a single file some actions (unit / individual actions) among all types of actions that can be executed on the electronic device (200).

[0115] In one embodiment, the program may include a set of actions. The program may be for setting and / or executing a set of actions related to a task (e.g., execution assistance, automatic execution, simple execution).

[0116] In one embodiment, the program may include UI information. The program may be configured to provide a user interface associated with at least one action.

[0117] According to one embodiment, the processor (210) can obtain a program generated (e.g., programmed) by the third AI model by inputting at least a portion of the description and device information of the electronic device (200) into the third AI model. For example, the processor (210) can generate a program for an action set by inputting at least a portion of the description for at least one task and device information of the electronic device (200) into the third AI model (e.g., the MLLM model (614) of FIG. 6 or the MLLM model (1014) of FIG. 10).

[0118] In one embodiment, the program may be generated in the form of an executable file. In one embodiment, the program may be generated in the form of an application requiring installation (e.g., a mobile application package such as an '.apk' extension file for Android OS or an '.ipa' extension file for iOS). In one embodiment, the program may also be generated in the form of code (e.g., script data) that can be directly executed without installation.

[0119] In one embodiment, the program may be a personalized (or customized) result that converts (or adapts) at least one operation required to perform the task indicated by the content into at least one action suitable for the user's electronic device (200).

[0120] According to one embodiment, the processor (210) may display a user interface related to the program via the display (230).

[0121] According to one embodiment, the processor (210) may display a user interface related to the program based on UI information included in the program. For example, the UI information may be generated (or provided) by at least one generative AI model (e.g., the MLLM model (614) of FIG. 6 and the MLLM model (1014) of FIG. 10).

[0122] In one embodiment, the user interface associated with the program may include an object (e.g., an icon) corresponding to the program. The object may be for automatically executing a set of actions (at least one action) set by the program in batches. The object may be a component of a user interface (UI) or UI screen, and may also be expressed by other terms such as an icon, button, function button, soft key, menu, indicator, graphic element, visual element, UI element, or UI area.

[0123] In one embodiment, the object may be for executing the program. For example, the object may be for executing a set of actions (at least one action) set by the program. For example, the object may be for automatically executing multiple actions within the set of actions set by the program in batches.

[0124] According to one embodiment, a user interface associated with a program may include a UI screen for each action. The UI screen for each action may be associated with at least some of a set of actions set by the program. For example, if the set of actions includes multiple actions, UI screens for some of the multiple actions may be displayed as the program is executed. For example, a UI screen may be displayed for a first action that is executed through user interaction (e.g., an action that requires user input). For example, a UI screen may not be displayed for a second action that is executed without user interaction (e.g., an action that does not require user input).

[0125] According to one embodiment, the processor (210) may execute the program based on a first user input (e.g., a touch on an icon) for an object corresponding to the program.

[0126] According to one embodiment, the processor (210) may, based on a second user input for an object corresponding to the program (e.g., a long touch on an icon), display, via the display (230), a user interface element for at least one of a first option for editing an action set by the program or a second option for sharing the program with another user.

[0127] According to one embodiment, the processor (210) may provide a notification indicating the execution status of a task through the display (230). For example, the notification may include a user interface element (or message) indicating at least one of information regarding whether a task has been initiated, whether a task has been completed, a task progress percentage, and whether a task or action has been executed on an external electronic device.

[0128] In one embodiment, the processor (210) may display a description of at least one task via the display (230). The processor (210) may modify the description based on user input. The processor (210) may update a program for an action set based on the modified description.

[0129] According to one embodiment, a user of the electronic device (200) can share a program for a set of actions and / or a description corresponding to the program with a user of an external electronic device (e.g., electronic device (102, 104) of FIG. 1).

[0130] According to one embodiment, the processor (210) may transmit a description of at least one task and / or a program for a set of actions required to perform a task to an external electronic device of another user (e.g., the electronic device (102, 104) of FIG. 1) for sharing. The processor (210) may transmit, based on a user input, a description of at least one task and / or a program for a set of actions to the external electronic device of another user (e.g., the electronic device (102, 104) of FIG. 1) via the communication circuit (240). The processor (210) may receive, via the communication circuit (240), a notification regarding the performance status of the task (e.g., whether the task or action has been performed on the external electronic device). The electronic device (200) may display the notification via the display (230).

[0131] According to one embodiment, a program (or at least a portion of a set of actions of said program) may be configured to execute automatically when a specified execution condition is satisfied.

[0132] According to one embodiment, the processor (210) may store information about the execution conditions of the program (e.g., conditions specified for location, time, action, etc.) in the memory (220). The processor (210) may execute the program and / or at least one action set by the program based on satisfaction of the execution conditions.

[0133] In one embodiment, the execution deadline of the task (or the validity period of the program) may be set based on the content representing the task to be performed.

[0134] According to one embodiment, the processor (210) may store information about a task's execution deadline and notification conditions in the memory (220). Based on the satisfaction of the notification conditions, the processor (210) may display, through the display (230), a first notification notifying of the expiration of the execution deadline or a second notification guiding that the program needs to be executed before the expiration of the execution deadline.

[0135] In one embodiment, a deadline for performing a task (or a program's validity period) may be set based on the content.

[0136] According to one embodiment, when a deadline for performing a task (or a program's validity period) is set, the processor (210) registers information about notification conditions (e.g., information about reminders, to-dos, schedules, etc.) to an application related to notification (e.g., a schedule application), thereby allowing the notification and / or task execution to be managed through the application.

[0137] According to one embodiment, the processor (210) may perform an integrity check on the program (or at least one action included in the program) when executing the program. For example, after the program is generated, the device information (e.g., application information) of the electronic device (200) may change. For example, after the program is generated, a previous application may be deleted, a new application may be added, or the application usage history may be changed. For example, there may be a difference between the first device information at the time of generating the program and the second device information at the time of executing the program (or the action of the program). In this case, the processor (210) may determine whether the program (or action) can be executed through an integrity check. If the program (or action) cannot be executed, the processor (210) may update the program or regenerate the program based on the program and / or a description corresponding to the program. For example, the processor (210) can update the program or generate a new program by inputting the program (or a description corresponding to the program) and the second device information at the time of execution into at least one generative AI model (e.g., the MLLM model (614) of FIG. 6 or the MLLM model (1014) of FIG. 10).

[0138] In one embodiment, the processor (210) may generate a program for a set of actions based on personal information (e.g., age, application usage history, health status) of the user of the electronic device (200). For example, the personal information may be stored in the memory (220) or stored in an external electronic device (e.g., server (108) of FIG. 1) in conjunction with a user account.

[0139] According to one embodiment, the processor (210) may further input personal information into at least one generative AI model (e.g., the MLLM model (614) of FIG. 6 or the MLLM model (1014) of FIG. 10) to generate a program that reflects personal information.

[0140] For example, a program's action may be selected based on age. For example, if the user is a minor, the program's action may be selected to ensure that the application, content, or function complies with the age restriction. For example, a program's action may be selected based on application usage history. For example, if there are multiple applications that provide similar functionality, the program's action may be selected based on application usage history. For example, a program's action may be selected based on health status. For example, for a user with visual impairment, an action in a program that uses an auditory user interface may be selected for accessibility reasons.

[0141] According to one embodiment, the processor (210) may add a program for a set of actions to a user routine, thereby causing a task corresponding to the program to be performed repeatedly (e.g., periodically or whenever a specified condition is satisfied) based on a routine execution condition.

[0142] According to one embodiment, the processor (210) may receive a user request (e.g., voice, text) using an intelligent assistant. The processor (210) may determine a target application related to the user request from among a plurality of applications installed on the electronic device (200). The processor (210) may display an object corresponding to the program using the intelligent assistant and the target application. The object may be for executing the program. In response to a user input for the object, the processor (210) may execute an action set (at least one action) set by the program in conjunction with the target application. As the action set is executed, the electronic device (200) may display the execution status of the task through the user interface of the intelligent assistant.

[0143] According to one embodiment, at least one task of the description may include a first task and a second task. The set of actions set by the program (or included in the program) may include a first action provided through a first application and a second action provided through a second application. The first application and the second application may be different from each other. The first task may be combined with the first action (e.g., an action requiring user input) that is executed through interaction with a user of the electronic device (200). The second task may be combined with the second action (e.g., an action that does not require user input) that is executed without interaction with a user of the electronic device (200).

[0144] In one embodiment, the processor (210) may control the display (230) to highlight a user interface element of the first application corresponding to the first action as the program is executed. The processor (210) may execute the first action based on a user input to the user interface element.

[0145] According to one embodiment, the processor (210) may perform an operation of obtaining a description based on content, and / or an operation of generating a program based on the content, using at least one generative AI model.

[0146] According to one embodiment, the processor (210) can obtain a description generated based on content using a first AI model (e.g., at least one of the frame analysis AI model (611) of FIG. 6 or the action parsing AI model (1011) of FIG. 10) and / or a second AI model (e.g., at least one of the LVM model (612) of FIG. 6, the LLM model (613) or the LLM model (1013) of FIG. 10). The processor (210) can obtain a program generated based on the content using a third AI model (e.g., the MLLM model (614) of FIG. 6 or the MLLM model (1014) of FIG. 10).

[0147] In one embodiment, at least some of the first AI model, the second AI model, or the third AI model may be the same AI model (e.g., a unified AI model).

[0148] According to one embodiment, at least some of the first AI model, the second AI model, and the third AI model may be provided within the electronic device (200) in the form of on-device AI models.

[0149] According to one embodiment, at least some of the first AI model, the second AI model, or the third AI model may be provided from outside the electronic device (200) (e.g., the server (108) of FIG. 1).

[0150] In one embodiment, some of the first AI model, the second AI model, and the third AI model may be provided internally in the form of on-device AI models of the electronic device (200). Others of the first AI model, the second AI model, and the third AI model may be provided externally of the electronic device (200) (e.g., the server (108) of FIG. 1).

[0151] According to one embodiment, the electronic device (200) or the processor (210) of the electronic device (200) may interact with at least one external electronic device (e.g., the electronic device (102, 104) of FIG. 1) to perform a task indicated by the content.

[0152] According to one embodiment, at least one action among a set of actions (at least one action) set by the program may be executed (or provided) through an external electronic device (e.g., an Internet of Things device) (e.g., the electronic device (102) of FIG. 1) connected to the electronic device (200) via a designated network (e.g., a user's home network). A notification regarding the performance status of a task related to the at least one action may be displayed on the external electronic device.

[0153] According to one embodiment, at least one action within the set of actions set by the program can be executed (or provided) via a wearable electronic device or an XR (extended reality) device connected to the electronic device (200) via short-range communication.

[0154] According to one embodiment, content used for performing a task may be collected through an external electronic device (e.g., an Internet of Things device, a wearable electronic device, an XR device) (e.g., the electronic device (102) of FIG. 1) and transmitted to the electronic device (200). According to one embodiment, the electronic device (200) may transmit setting values ​​required for executing an action of the external electronic device to the external electronic device, and the result of an action executed by the external electronic device according to the setting values ​​may be sent to the electronic device (200). According to one embodiment, the execution status of the corresponding task may be displayed through the external electronic device. According to one embodiment, when information collected from the external electronic device satisfies a specified condition, execution of the task may be automatically initiated.

[0155] According to one embodiment, a system including an electronic device (200) may include an AI hub (or AI platform) structure in which at least some modules or some sub-components of the modules are driven by an Always-on low-power AI processor (or engine). The electronic device (200) may intelligently connect and monitor in real time not only various applications, functions, input / output interfaces, and power elements of the electronic device (200) but also wearable electronic devices (e.g., smart watches, smart rings), XR devices, Internet of Things devices (e.g., home appliances, access points, speakers), cloud servers, and automobiles connected (or communication-connected) to the electronic device (200) via a network through the AI ​​hub structure, thereby executing at least one action (or at least one operation) of the present disclosure.

[0156] According to one embodiment, the processor (210) of the electronic device (200) may refer to a plurality of processors that collectively perform a plurality of actions (or operations) by dividing them among the processors. The modules of the present disclosure may include software or hardware driven by an application processor or a dedicated processor.

[0157] According to one embodiment, the electronic device (200) may include a foldable device or a rollable device configured such that the size of the screen exposed on the front of the electronic device (200) changes as a portion of the housing moves. For example, tasks may be performed by different sets of actions depending on the folding or rolling state of the screen. According to one embodiment, the electronic device (200) may be a foldable device (e.g., a foldable smart phone). In this case, a first description and / or a first program corresponding to (suitable for) a case in which the foldable device is in a folded state, and a second description and / or a second program corresponding to (suitable for) a case in which the foldable device is in an unfolded state may be generated.

[0158] According to one embodiment, the electronic device (200) may be a multi-foldable device. The multi-foldable device may include two or more hinge assemblies to divide the flexible display into three or more regions, and may be of a type (e.g., e type, G type, z type, book type) that can be folded on the left and right sides based on the center region. According to one embodiment, a description and / or program for at least one operation required to perform a task may be provided through different user interfaces depending on the folding state or screen size of the multi-foldable device. The task and the action set (at least one action or a series of actions) for performing the task may be generated differently to reflect usage conditions (size of the screen used, type of screen used, orientation of the multi-foldable device, grip direction of the user) that vary depending on the folding state of the multi-foldable device. A specific task or action may be generated so that it can be executed only under a specific folding condition (screen usage condition) of the foldable device. For example, a program may be executed based on a determination of whether the folding condition of the foldable device is a specified condition. For example, the program may be executed using a first action set if the folding condition of the foldable device is a first condition, and may be executed using a second action set if the folding condition is a second condition. The first and second action sets may differ in at least one of the application, camera, or screen area used. For example, the first action set may be executed using a first camera (e.g., a front camera) of the electronic device (200), and the second action set may be executed using a second camera (e.g., a rear camera) of the electronic device (200). For example, the first action set may be executed using a fingerprint sensor, and the second action set may be executed using a camera.

[0159] FIG. 3 is a flowchart illustrating an operation method of an electronic device (200) according to one embodiment, and a method in which the electronic device (200) supports action execution.

[0160] The operations illustrated in FIG. 3 may be performed sequentially, but are not necessarily performed sequentially. For example, the order of the operations may be changed, and at least two operations may be performed in parallel. In some embodiments, some of the illustrated operations may be omitted, some operations may be combined, some operations may be reordered, or other operations may be added.

[0161] Referring to FIG. 3, an operating method of an electronic device (200) according to one embodiment may include operations 310, 320, 330, 340, and 350.

[0162] According to one embodiment, at operation 310, the electronic device (200) (e.g., processor (210)) may obtain content indicating a task to be performed.

[0163] In one embodiment, the content may include various types of content (e.g., text, voice, audio, images (still images), videos, multimedia). For example, the content may include information about a task and / or at least one task required to perform the task.

[0164] In one embodiment, the content representing the task to be performed may be selected based on user input of the electronic device (200).

[0165] According to one embodiment, the content representing the task to be performed may be generated by an external electronic device (e.g., electronic device (102, 104) of FIG. 1) that is an electronic device of another user and transmitted to the electronic device (200).

[0166] In one embodiment, the content representing the task to be performed may be received through a conversation session (e.g., a conversation session for a chat app) in which at least one message is exchanged between a user of the electronic device (200) and another user of an external electronic device (e.g., the electronic devices (102, 104) of FIG. 1 ). For example, the conversation session may correspond to a communication channel between the electronic device (200) and the external electronic device.

[0167] In one embodiment, the content representing the task to be performed may be received by an intelligent assistant interacting with the user of the electronic device (200) via voice.

[0168] According to one embodiment, the content representing the task to be performed may be generated through handwriting input. For example, the electronic device (200) may receive text corresponding to the handwriting input through the display (230). The text may be written by the user using a stylus pen (or electronic pen). The text may be text related to the task to be performed (e.g., create a morning routine) or text related to at least one task (e.g., play XX music in the morning). The electronic device (200) may identify the task or at least one task related to the text using a generative AI model. The operation of identifying the task or at least one task from the handwriting input may be initiated (or triggered) based on a designated input. For example, the designated input may be an input of drawing a stroke of a designated shape around the handwriting input.

[0169] According to one embodiment, in operation 320, the electronic device (200) (e.g., the processor (210)) may obtain a description (or specification) of at least one operation required to perform the task based on content representing the task to be performed. For example, the description may correspond to a description (or explanation) of a work procedure required to perform the task.

[0170] In one embodiment, the description may include data regarding at least some of the multiple operations necessary for performing the task or constituting the task. For example, the description may include text describing at least one unit operation (or individual operation) necessary for performing the task. For example, the description may include text describing at least some of the unit operations necessary for performing the task, regardless of the order. For example, the description may include text describing at least some of the unit operations having continuity in a stepwise or sequential manner.

[0171] According to one embodiment, the electronic device (200) can obtain a description generated based on the content using at least one generative AI model.

[0172] According to one embodiment, the electronic device (200) can obtain a description generated based on the content by inputting the content representing the task into at least one generative AI model.

[0173] According to one embodiment, the electronic device (200) can obtain a description generated based on content using the first AI model and the second AI model.

[0174] For example, the electronic device (200) can obtain (e.g., extract) an image data set (an image data set related to at least one operation of a task) generated by the first AI model by inputting a video corresponding to content (e.g., a guide video) into the first AI model (e.g., a frame analysis AI model (611) of FIG. 6). The electronic device (200) can obtain a description generated by the second AI model by inputting the image data set into the second AI model (e.g., an LVM model (612) and / or an LLM model (613) of FIG. 6).

[0175] For example, image embedding data (e.g., 622 in FIG. 6) extracted from a video and text embedding data (e.g., 623 in FIG. 6) extracted through the LVM model (612) can be used as inputs to the LLM model (613) to obtain a description. The electronic device (200) can obtain a description by inputting the image embedding data and the text embedding data into the LLM model (613).

[0176] For example, the electronic device (200) can extract a text data set by inputting text corresponding to content (e.g., guide text) into a first AI model (e.g., the action parsing AI model (1011) of FIG. 10). The electronic device (200) can obtain a description generated by the second AI model by inputting the text data set into a second AI model (e.g., the LLM model (1013) of FIG. 10).

[0177] In one embodiment, the first AI model and the second AI model used to obtain the description may be the same AI model. The electronic device (200) can obtain a description generated based on content using the same AI model.

[0178] According to one embodiment, in operation 330, the electronic device (200) (e.g., processor (210)) may extract device information of the electronic device (200). For example, the electronic device (200) may extract device information from memory (220).

[0179] According to one embodiment, the device information of the electronic device (200) may include at least some of application information (e.g., information about version, category, type, application identifier, application UI, application function, etc.) for application(s) installed in the electronic device (200), software information (e.g., information about OS (operating system) version, UI (user interface) / UX (user experience) version, etc.), or hardware information (e.g., information about device type, device identifier, specifications, camera, biometric sensor, etc.). The device information may include, for example, status information of the electronic device (200) (e.g., remaining battery, whether charging, type of connected network, whether the screen is locked, location). The electronic device (200) may also obtain (e.g., receive) device information of an external electronic device (e.g., the electronic devices (102, 104) of FIG. 1) (e.g., an Internet of Things device connected to a user's home network, a wearable electronic device registered to a user account of the user).

[0180] According to one embodiment, in operation 340, the electronic device (200) (e.g., processor (210)) may generate a program (e.g., an action executable file) for an action set to be executed in the electronic device (200) based on the description of at least one task obtained through operation 320 and the device information of the electronic device (200) extracted through operation 330. The action set may include at least one action.

[0181] According to one embodiment, the electronic device (200) (e.g., processor (210)) may generate a program (e.g., an action execution file) for a set of actions (at least one action) to be executed on the electronic device (200) based further on device information of the external electronic device (e.g., an Internet of Things device, a wearable electronic device of a user).

[0182] According to one embodiment, the action set may include at least one action (or function) executable by the electronic device (200).

[0183] According to one embodiment, the action set may include at least one action (or function) that the electronic device (200) can execute in conjunction with a designated external electronic device (e.g., electronic device (102, 104) of FIG. 1) (e.g., an Internet of Things device connected to the user's home network, a wearable electronic device registered to the user's user account).

[0184] According to one embodiment, the action set may include a series of action steps configured by a generative AI model that are not provided by the manufacturer in the default setting state of the electronic device (200).

[0185] According to one embodiment, the action set may include an action for installing an application. For example, if a first application capable of executing an action required for performing a task is not installed on the electronic device (200), an action for installing the first application or a second application corresponding to the first application may be included in the action set. For example, the processor (210) may display a user interface element guiding the installation of the first application or the second application through the display (230). The processor (210) may install the first application or the second application on the electronic device (200) in response to a user input (e.g., an installation approval input) for the interface element.

[0186] According to one embodiment, the program may be for setting and / or executing a set of actions (at least one action) to be executed on the electronic device (200).

[0187] In one embodiment, the program may include a set of actions. The program may be for setting and / or executing a set of actions related to a task (e.g., execution assistance, automatic execution, simple execution).

[0188] According to one embodiment, the program may include UI information (e.g., object information, UI element information, and UI screen information for each action) for performing a task. The program may be for providing a user interface related to at least one action.

[0189] According to one embodiment, the electronic device (200) can obtain a program generated (e.g., programmed) by the third AI model by inputting at least a portion of the description obtained through operation 320 and the device information of the electronic device (200) obtained through operation 330 into the third AI model. For example, the electronic device (200) can generate a program for an action set by inputting at least a portion of the description for at least one task and the device information of the electronic device (200) into the third AI model (e.g., the MLLM model (614) of FIG. 6 or the MLLM model (1014) of FIG. 10).

[0190] In one embodiment, the program may be generated in the form of an executable file. In another embodiment, the program may be generated in the form of an application that requires installation (e.g., a mobile application package such as an '.apk' extension file for Android OS or an '.ipa' extension file for iOS). In another embodiment, the program may also be generated in the form of code that can be directly executed without installation (e.g., script data).

[0191] According to one embodiment, at operation 350, the electronic device (200) (e.g., processor (210)) may display a user interface related to the program via the display (230).

[0192] According to one embodiment, the electronic device (200) may display a user interface related to the program based on UI information included in the program. The UI information may be generated (or provided) by at least one generative AI model (e.g., the MLLM model (614) of FIG. 6 and the MLLM model (1014) of FIG. 10).

[0193] In one embodiment, the user interface associated with the program may include an object (e.g., an icon) corresponding to the program. The object may be a component of a user interface (UI) or UI screen, and may also be expressed by other terms such as an icon, button, function button, soft key, menu, graphic element, visual element, UI element, or UI area.

[0194] In one embodiment, the object may be for executing the program. For example, the object may be for executing a set of actions (at least one action) set by the program. For example, the object may be for automatically executing at least one action within the set of actions set by the program. For example, the object may be for automatically executing multiple actions within the set of actions set by the program in batches.

[0195] According to one embodiment, a user interface associated with a program may include a UI screen for each action. The UI screen for each action may be associated with at least some of a set of actions set by the program. For example, if the set of actions includes multiple actions, UI screens for some of the multiple actions may be displayed as the program is executed. For example, a UI screen may be displayed for a first action that is executed through user interaction (e.g., an action that requires user input). For example, a UI screen may not be displayed for a second action that is executed without user interaction (e.g., an action that does not require user input).

[0196] According to one embodiment, the electronic device (200) can execute the program based on a first user input (e.g., a touch on an icon) to an object corresponding to the program.

[0197] According to one embodiment, the electronic device (200) may, based on a second user input for an object corresponding to the program (e.g., a long touch on an icon), display, via the display (230), a user interface element for at least one of a first option for editing an action set by the program or a second option for sharing the program with another user.

[0198] According to one embodiment, the electronic device (200) may provide a notification indicating the execution status of a task through the display (230). For example, the notification may include a user interface element (or message) indicating at least one of information regarding whether a task has been initiated, whether a task has been completed, a task progress percentage, and whether a task or action has been executed on an external electronic device.

[0199] According to one embodiment, the electronic device (200) can display a description of at least one task via the display (230). The electronic device (200) can modify the description based on user input. The electronic device (200) can update a program for an action set based on the modified description.

[0200] According to one embodiment, a user of the electronic device (200) can share a program for a set of actions and / or a description corresponding to the program with a user of an external electronic device (e.g., electronic device (102, 104) of FIG. 1).

[0201] According to one embodiment, the electronic device (200) may transmit a description of at least one task and / or a program for a set of actions required to perform a task to an external electronic device of another user (e.g., the electronic device (102, 104) of FIG. 1) for sharing. Based on a user input, the electronic device (200) may transmit at least a portion of the description of at least one task and / or the program for a set of actions to the external electronic device of another user (e.g., the electronic device (102, 104) of FIG. 1) via the communication circuit (240). The electronic device (200) may receive a notification regarding the performance status of the task (e.g., whether the task or action has been performed on the external electronic device) from the external electronic device via the communication circuit (240). The electronic device (200) may display the notification via the display (230).

[0202] According to one embodiment, a program (or at least a portion of a set of actions of said program) may be configured to execute automatically when a specified execution condition is satisfied.

[0203] According to one embodiment, the electronic device (200) may store information about the execution conditions of a program (e.g., conditions specified for location, time, action, etc.) in the memory (220). The electronic device (200) may execute the program and / or at least one action set by the program based on satisfaction of the execution conditions.

[0204] In one embodiment, the execution deadline of the task (or the validity period of the program) may be set based on the content representing the task to be performed.

[0205] According to one embodiment, when a deadline for performing a task (or a program's validity period) is set, the electronic device (200) registers information about notification conditions (e.g., information about reminders, to-dos, schedules, etc.) in an application related to notifications (e.g., a schedule application), thereby allowing the notification and / or task performance to be managed through the application.

[0206] According to one embodiment, the electronic device (200) may store information about a task's execution deadline and notification conditions in the memory (220). Based on the satisfaction of the notification conditions, the electronic device (200) may display a first notification notifying the expiration of the execution deadline, or a second notification guiding the necessity of executing the program before the expiration of the execution deadline.

[0207] According to one embodiment, the electronic device (200) may perform an integrity check on the program (or at least one action included in the program) when executing the program. For example, device information (e.g., application information) of the electronic device (200) may change after the program is created. For example, after the program is created, a previous application may be deleted, a new application may be added, or application usage history may be changed. For example, there may be a difference between the first device information at the time of program creation and the second device information at the time of execution of the program (or the action of the program). In this case, the electronic device (200) may determine whether the program (or action) can be executed through an integrity check. If the program (or action) cannot be executed, the electronic device (200) may update the program or regenerate the program based on the program and / or a description corresponding to the program. For example, the electronic device (200) can update the program or create a new program by inputting the program (or a description corresponding to the program) and the second device information at the time of execution into at least one generative AI model (e.g., the MLLM model (614) of FIG. 6 or the MLLM model (1014) of FIG. 10).

[0208] In one embodiment, the electronic device (200) may generate a program for a set of actions based on personal information of the user of the electronic device (200), such as age, application usage history, and health status. For example, the personal information may be stored in the memory (220) or stored in an external electronic device (e.g., the server (108) of FIG. 1) in conjunction with a user account.

[0209] According to one embodiment, the electronic device (200) may generate a program reflecting personal information by additionally inputting personal information into at least one generative AI model (e.g., the MLLM model (614) of FIG. 6 or the MLLM model (1014) of FIG. 10).

[0210] For example, a program's action may be selected based on age. For example, if the user is a minor, the program's action may be selected to ensure that the application, content, or function complies with the age restriction. For example, a program's action may be selected based on application usage history. For example, if there are multiple applications that provide similar functionality, the program's action may be selected based on application usage history. For example, a program's action may be selected based on health status. For example, for a user with visual impairment, an action in a program that uses an auditory user interface may be selected for accessibility reasons.

[0211] According to one embodiment, the electronic device (200) can cause a task corresponding to the program to be performed repeatedly (e.g., periodically or whenever a specified condition is satisfied) based on a routine execution condition by adding a program for an action set to a user routine.

[0212] According to one embodiment, an electronic device (200) can receive a user request (e.g., voice, text) using an intelligent assistant. The electronic device (200) can determine a target application related to the user request from among a plurality of applications installed on the electronic device (200). The electronic device (200) can display an object corresponding to the program using the intelligent assistant and the target application. The object may be for executing the program. In response to a user input for the object, the electronic device (200) can execute an action set (at least one action) set by the program in conjunction with the target application. As the action set is executed, the electronic device (200) can display the execution status of the task through the user interface of the intelligent assistant.

[0213] According to one embodiment, at least one task of the description may include a first task and a second task. The set of actions set by the program (or included in the program) may include a first action provided through a first application and a second action provided through a second application. The first application and the second application may be different from each other. The first task may be combined with the first action (e.g., an action requiring user input) that is executed through interaction with a user of the electronic device (200). The second task may be combined with the second action (e.g., an action that does not require user input) that is executed without interaction with a user of the electronic device (200).

[0214] According to one embodiment, the electronic device (200) may highlight a user interface element of the first application corresponding to the first action as the program is executed. The electronic device (200) may execute the first action based on a user input to the user interface element.

[0215] According to one embodiment, the electronic device (200) can perform a task indicated by the content by interfacing with at least one external electronic device (e.g., the electronic device (102, 104) of FIG. 1).

[0216] According to one embodiment, at least one action among a set of actions (at least one action) set by the program may be executed (or provided) through an external electronic device (e.g., an Internet of Things device) (e.g., the electronic device (102) of FIG. 1) connected to the electronic device (200) via a designated network (e.g., a user's home network). A notification regarding the performance status of a task related to the at least one action may be displayed on the external electronic device.

[0217] According to one embodiment, the electronic device (200) (e.g., processor (210)) can perform at least some of the operations illustrated in FIG. 3 using at least one generative AI model, for example, operation 320 of obtaining a description based on content, and / or operation 340 of generating a program based on the content.

[0218] According to one embodiment, the electronic device (200) can obtain a description generated based on content using a first AI model (e.g., at least one of the frame analysis AI model (611) of FIG. 6 or the action parsing AI model (1011) of FIG. 10) and / or a second AI model (e.g., at least one of the LVM model (612) of FIG. 6, the LLM model (613) or the LLM model (1013) of FIG. 10). The electronic device (200) can obtain a program generated based on the content using a third AI model (e.g., the MLLM model (614) of FIG. 6 or the MLLM model (1014) of FIG. 10).

[0219] In one embodiment, at least some of the first AI model, the second AI model, or the third AI model may be the same AI model (e.g., a unified AI model).

[0220] According to one embodiment, at least some of the first AI model, the second AI model, and the third AI model may be provided within the electronic device (200) in the form of on-device AI models.

[0221] According to one embodiment, at least some of the first AI model, the second AI model, or the third AI model may be provided from outside the electronic device (200) (e.g., the server (108) of FIG. 1).

[0222] In one embodiment, some of the first AI model, the second AI model, and the third AI model may be provided internally in the form of on-device AI models of the electronic device (200). Others of the first AI model, the second AI model, and the third AI model may be provided externally of the electronic device (200) (e.g., the server (108) of FIG. 1).

[0223] FIG. 4 is an example of a user interface related to a process (e.g., operation 340 of FIG. 3) in which an electronic device (200) generates a program using a video according to one embodiment.

[0224] According to one embodiment, the electronic device (200) can generate a program based on a video. The electronic device (200) can obtain the program generated based on the video by inputting the video into at least one generative AI model. The program may be for performing a task (e.g., issuing a family relationship certificate). The program may be for setting and / or executing a set of actions related to the task (e.g., execution support, automatic execution, simple execution).

[0225] Referring to FIG. 4, the electronic device (200) can display UI screens, such as a first screen (410), a second screen (420), a third screen (430), a fourth screen (440), and a fifth screen (450), through a display (230). The first screen (410), the second screen (420), the third screen (430), the fourth screen (440), and the fifth screen (450) may be for generating a program using a video.

[0226] In the embodiment of Fig. 4, the number of screens is exemplified as five, but the number of screens is not limited to five. For example, additional screens may be created, or the number of screens may be reduced.

[0227] The first screen (410) and the second screen (420) are for explaining an operation of obtaining a video corresponding to content representing a task to be performed (e.g., operation 310 of FIG. 3).

[0228] In one embodiment, the video may be selected based on user input. For example, the video may be a video (411) (e.g., a guide video) representing a specific task (e.g., issuing a family relationship certificate).

[0229] According to one embodiment, the electronic device (200) may display a first screen (410). The first screen (410) may display a video (411) indicating a task to be performed (e.g., a video for a family relationship certificate issuance task) and / or a UI element (e.g., a play button (412)).

[0230] According to one embodiment, the electronic device (200) may detect a user input (413) (e.g., a long touch on a video playback area) on the first screen (410). When the electronic device (200) detects the user input (413) on the first screen (410), the electronic device (200) may display a second screen (420). A menu window (421) (or options window) regarding a simple action function may appear on the second screen (420). The menu window (421) may include an action setting menu.

[0231] According to one embodiment, the electronic device (200) may determine the displayed video (411) as a video representing a task to be performed based on a user input (e.g., touch) that selects the action setting menu. The electronic device (200) may obtain (e.g., extract, load, receive, download) the video. For example, the video may be a video (411) representing a specific task (e.g., issuing a family relationship certificate). For example, the video may include at least one of a source video (or video file) including a plurality of frames, a source video played and / or displayed on the electronic device (200), a video including the source video and UI elements together, or a recorded video of screens of the display (230).

[0232] The third screen (430) and the fourth screen (440) are for explaining an operation of generating a program based on a video (e.g., operation 340 of FIG. 3).

[0233] According to one embodiment, the electronic device (200) may initiate (or trigger) program generation in response to a user input (e.g., touch) selecting the action setting menu in the menu window (421) of the second screen (420). While the program is being generated, a third screen (430) may be displayed. The third screen (430) may include a notification window (431). The notification window (431) may display a message indicating that the program is being generated (e.g., "Creating action...").

[0234] According to one embodiment, the electronic device (200) can obtain (e.g., extract from memory (220)) device information (e.g., at least one of application information, software information, or hardware information) of the electronic device (200).

[0235] According to one embodiment, the electronic device (200) may generate a program for performing a task suitable for the electronic device (200) based on the video (411) and the device information of the electronic device (200). For example, the electronic device (200) may obtain a program generated based on the video (411) and the device information by inputting at least a portion of the video (411) and the device information into at least one generative AI model (e.g., MLLM (614) of FIG. 6, MLLM (1014) of FIG. 10).

[0236] In some embodiments, additional information about the video (411) (e.g., link information, download path information, related website information) may be additionally input to the at least one generative AI model for program generation. In such a case, the at least one generative AI model may provide a program generated based on at least a portion of the video (411), the device information, and the additional information about the video (411).

[0237] In one embodiment, the program may be for performing a specific task (e.g., issuing a family relationship certificate). In relation to the task, the program may be for setting and / or executing specific actions (e.g., unit / individual actions for the family relationship certificate issuance task) to be automatically executed on the electronic device (200).

[0238] According to one embodiment, the electronic device (200) may display a fourth screen (440) when the creation of the program is completed. The fourth screen (440) may include a notification window (441). The notification window (441) may display a message notifying the completion of the creation of the program and an input field for requesting the user to enter the name of the program.

[0239] The fifth screen (450) is intended to explain an operation (e.g., operation 350 of FIG. 3) for displaying a user interface related to the above program.

[0240] According to one embodiment, the electronic device (200) can display an object (451) (e.g., an icon) corresponding to the program through the fifth screen (450).

[0241] In one embodiment, the object (451) (e.g., an icon) may be included in a program. For example, the object (451) may be generated (or provided) by at least one generative AI model (e.g., the MLLM model (614) of FIG. 6, the MLLM model (1014) of FIG. 10).

[0242] According to one embodiment, the object (451) may be for performing a task. The object (451) may be for executing the program (or an action set corresponding to the program). For example, when the electronic device (200) detects a user input (e.g., a touch) to the object (451), it may perform a task corresponding to the program. For example, when the electronic device (200) detects a user input (e.g., a touch) to the object (451), it may execute (or trigger) an action set (or at least one action) set by the program. For example, when the electronic device (200) detects a user input (e.g., a touch) to the object (451), it may automatically execute an action set (at least one action) set by the program at once. The file name of the program may correspond to a name entered in an input field in the notification window (441).

[0243] FIG. 5 is an example of a user interface related to a process (e.g., operation 350 of FIG. 3) in which an electronic device (200) according to one embodiment displays a user interface related to a program.

[0244] According to one embodiment, the electronic device (200) may display a user interface related to the program based on UI information (e.g., object information, action-specific UI screen information) included in the program. For example, the UI information may be generated (or provided) by at least one generative AI model (e.g., the MLLM model (614) of FIG. 6, the MLLM model (1014) of FIG. 10).

[0245] According to one embodiment, the electronic device (200) may display a user interface (e.g., object (511) of FIG. 5) related to a program (or at least one action set by the program) corresponding to a task as the program is generated (e.g., operation 340 of FIG. 3).

[0246] According to one embodiment, the electronic device (200) may display a user interface (e.g., action-specific UI screens (530, 540, 550) of FIG. 3) related to a program corresponding to a task (or at least one action set by the program) as the program is executed.

[0247] Referring to FIG. 5, the electronic device (200) can display UI screens, such as a first screen (510), a second screen (520), a third screen (530), a fourth screen (540), and a fifth screen (550), via the display (230). The number of screens illustrated in FIG. 5 is merely an example for illustration, and the number of screens is not limited thereto. For example, additional screens may be created, or the number of screens may be reduced.

[0248] According to one embodiment, the electronic device (200) may display a first screen (510). The first screen (510) may include an object (511) (e.g., an icon) corresponding to a program. The object (511) of FIG. 5 may correspond to the object (451) of FIG. 4.

[0249] According to one embodiment, the program may include UI information regarding an object (511). The electronic device (200) may display a user interface (e.g., object (511)) related to the program based on the UI information (e.g., object information) included in the program.

[0250] According to one embodiment, the electronic device (200) may display a second screen (520) when a user input (e.g., touch) occurs on an object (511) within the first screen (510). A menu window (521) (or options window) regarding a simple action function may appear on the second screen (520). The menu window (521) may include an action execution menu.

[0251] According to one embodiment, when a user input (e.g., a touch) selecting the action execution menu occurs, the electronic device (200) can automatically execute specific actions set by the program (e.g., unit / individual actions for a family relationship certificate issuance task). For example, the specific actions may include a first action, a second action, and a third action.

[0252] According to one embodiment, as the specific actions are automatically executed on the electronic device (200), the electronic device (200) may sequentially output a third screen (530) related to the first action, a fourth screen (540) related to the second action, and a fifth screen (550) related to the third action. For example, the third screen (530) may be a UI screen related to the action of executing a banking app. As the action of executing a banking app is executed, the third screen (530) may be displayed. The third screen (530) may include a button for issuing a financial certificate (531). As the action of clicking the button for issuing a financial certificate (531) is executed, a fourth screen (540) may be displayed. The fourth screen (540) may be a UI screen related to the action of clicking the button for issuing a financial certificate (531). The fourth screen (540) may include a certificate issuance menu (541) requiring the entry of personal information. When the certificate issuance menu (541) is selected (or the personal information entry action is executed), the fifth screen (550) may be displayed. The fifth screen (550) may be a UI screen related to the personal information entry action. The fifth screen (550) may display a notification window (542) containing text guiding the entry of personal information (e.g., "Please enter your name and resident registration number!") that may be displayed.

[0253] According to one embodiment, the electronic device (200) may prompt and / or wait for user input when necessary (e.g., while the fifth screen (550) is being output). For example, if the action set by the program includes an action that does not require user input or does not require display of a UI screen, the electronic device (200) may automatically execute the action (e.g., execute it in the background) without displaying the UI screen.

[0254] According to one embodiment, the program may include UI information regarding a third screen (530) related to the first action, a fourth screen (540) related to the second action, and a fifth screen (550) related to the third action. The electronic device (200) may display a user interface (e.g., a third screen (530), a fourth screen (540), and a fifth screen (550)) related to the program based on the UI information (e.g., action-specific UI screen information) included in the program.

[0255] Accordingly, a user of the electronic device (200) can easily learn by directly following specific actions related to a task (e.g., a family relationship certificate issuance task) indicated by a video. In addition, a user of the electronic device (200) can reduce human error by executing various actions related to a desired task in a simple and intuitive manner using a program appropriately created for the user's electronic device (200) without a complicated and difficult navigation procedure.

[0256] FIG. 6 is a diagram illustrating a process in which an electronic device (200) according to one embodiment generates a program from a video (e.g., a guide video) using at least one generative AI model.

[0257] Referring to FIG. 6, reference numerals 611, 612, 613, and 614 represent AI models, respectively. Reference numerals 601, 621, 622, 623, and 624 represent inputs to the AI ​​model, respectively. Reference numerals 631 and 632 represent outputs of the AI ​​model.

[0258] According to one embodiment, the AI ​​models (611, 612, 613, 614) illustrated in FIG. 6 may each be implemented in hardware and / or software to perform a predetermined function.

[0259] In the embodiment of FIG. 6, the number of AI models used to generate the program is exemplified as four, but the number of AI models is not limited to four. For example, additional AI models may be used, or at least some of the illustrated AI models (611, 612, 613, 614) may be integrated. For example, some of the illustrated AI models (611, 612, 613, 614) may be omitted.

[0260] According to one embodiment, the electronic device (200) of FIG. 2 may be configured to include at least some of the AI ​​models (611, 612, 613, 614) illustrated in FIG. 6. At least some of the AI ​​models (611, 612, 613, 614) may be included in the electronic device (200) (e.g., memory (230)) in the form of on-device AI models, but is not limited thereto. For example, at least some of the AI ​​models (611, 612, 613, 614) may be included in an external server (e.g., server (108) of FIG. 1). In some embodiments, the functions of at least some of the AI ​​models (611, 612, 613, 614) may be provided through a single integrated AI model.

[0261] According to one embodiment, at least some functions performed by an AI model within an external server (e.g., server (108) of FIG. 1) may be switched or replaced to be performed by an on-device AI model of the electronic device (200) depending on privacy and / or security levels.

[0262] According to one embodiment, as illustrated in FIG. 6, the electronic device (200) may perform a simple action function using at least one generative AI model among a frame analysis AI model (611), a large vision model (LVM) model (612), a large language model (LLM) model (613), or a multimodal large language model (MLLM) model (614). At least some of the AI ​​models (611, 612, 613, 614) illustrated in FIG. 6 may be generative AI models.

[0263] According to one embodiment, the electronic device (200) can generate a program (632) using a guide video (601). The guide video (601) can be composed of a plurality of frames (or images or UI screens). The guide video (601) can be about any task to be performed (e.g., a task of issuing a family relationship certificate). The guide video (601) can include a data set related to actions that a user must take in relation to the task (e.g., data on which application is executed, which button is clicked, and which text is input).

[0264] According to one embodiment, a guide video (601) and / or information about the guide video (601) (e.g., link information, download path information, related website information) may be input into a frame analysis AI model (611).

[0265] In one embodiment, the frame analysis AI model (611) may be an AI model trained for the purpose of extracting an image data set from a video.

[0266] According to one embodiment, the frame analysis AI model (611) can analyze frames of a guide video (601) and extract representative frames among the frames based on the analysis results.

[0267] For example, a representative frame may be a frame that requires user input (e.g., a frame that contains specific UI elements such as personal information input fields, agree / decline buttons, or authentication menus).

[0268] For example, a representative frame may be a dynamic frame in which a relatively large change has occurred compared to previous frames. When a screen scroll occurs within a video, frames representing screens before and after the scroll may have a relatively large difference. The frame analysis AI model (611) can infer the direction of movement by analyzing the difference and extract the screen after the scroll ends as a representative frame. When an application used within a video has changed, frames representing screens before and after the application change may have a relatively large difference. The frame analysis AI model (611) can extract the screen after the application change as a representative frame by recognizing the difference.

[0269] According to one embodiment, the frame analysis AI model (611) may output image embedding data (622) that embeds the extracted representative frames. The image embedding data (622) may be a result of converting the frames into a vector, which is a list of numbers that the LVM model (612) can understand. The image embedding data (622) may be an image data set corresponding to the representative frames.

[0270] In one embodiment, image embedding data (622) can be used as input to an LVM model (612).

[0271] In one embodiment, the LVM model (612) may be a generative AI model trained for the purpose of converting images into text or generating and / or providing text related to images.

[0272] According to one embodiment, the LVM model (612) can understand the architecture (e.g., UI shape, properties, categories, execution buttons within the UI, input fields, etc.) of each representative frame in the image embedding data (622), extract text from each representative frame, or generate text (e.g., keywords, phrases, or summary sentences) describing each representative frame. The LVM model (612) can output text embedding data (623) that embeds text for each representative frame.

[0273] According to one embodiment, image embedding data (622) extracted from a frame analysis AI model (611) and / or text embedding data (623) output from an LVM model (612) can be used as input to an LLM model (613).

[0274] In one embodiment, the LLM model (613) may be a generative artificial intelligence model trained for the purpose of generating and / or providing a description (631). The description (631) may be a description of at least one action.

[0275] In one embodiment, the LLM model (613) may be a language model trained on a large-scale data set. The LLM model (613) may perform various natural language processing tasks, such as text generation, translation, and question answering. The LLM model (613) may perform natural language processing using input image embedding data (622) and / or text embedding data (623), and output a description (631) based on the natural language processing results.

[0276] For example, the LLM model (613) can generate a description (631) for at least one task based on the image corresponding to each of the representative frames in the image embedding data (622) and / or the text describing each image in the text embedding data (623).

[0277] According to one embodiment, the LLM model (613) can generate a description (631) based on at least one of a guide video (601), at least one frame extracted from the guide video (601), image embedding data (622), or text embedding data (623).

[0278] For example, the LLM model (613) can perform UI information analysis for each representative frame based on the representative frame-by-frame image and / or representative frame-by-frame text extracted from the guide video (601). For example, the LLM model (613) can perform UI information analysis on which application should be operated, how the UI screen (e.g., text or icons, buttons, scrolls, etc. within the UI screen) is configured, which button or menu should be selected, or which function should be executed (or activated), etc., and can extract key UI information based on the UI information analysis result. The LLM model (613) can generate a description (631) using the extracted key UI information.

[0279] According to one embodiment, the description (631) may generate a description (631) for at least one operation required to perform the task indicated by the guide video (601). For example, the description (631) may correspond to a text data set that organizes the operations required to perform the task step by step or sequentially.

[0280] According to one embodiment, the description (631) may correspond to a text data set including natural language sentences for each of the step-by-step (or sequential) operations associated with the task.

[0281] In one embodiment, the description (631) and / or text embedding data (624) for the description (631) can be used as input to the MLLM model (614).

[0282] According to one embodiment, device information (621) of an electronic device (200) and / or text embedding data (624) for the device information (621) may be input into an MLLM model (614).

[0283] According to one embodiment, there may be a difference between the execution environment of the guide video (601) and the execution environment of the electronic device (200). The MLLM model (614) can understand the difference between the two execution environments by using the text embedding data (624) regarding the guide video (601) and the device information (621) of the electronic device (200), and can modify (or edit) the text embedding data (624) according to the difference.

[0284] According to one embodiment, the MLLM model (614) can use a mixture of different types of data (multimodal data).

[0285] According to one embodiment, the MLLM model (614) can extract at least one action from the description (631) and / or the description (631).

[0286] In one embodiment, the MLLM model (614) can understand the description (631) and / or at least one task described by the description (631), and map a set of actions (at least one action) to be included in a program corresponding to the at least one task. The MLLM model (614) can generate a program for the set of actions.

[0287] In one embodiment, the program may include at least one action (or a set of actions). The program may be capable of setting (or selecting) at least one action. The program may be configured to execute at least one action. The program may be configured to execute (e.g., support execution, automatic execution, or simple execution) at least one action for performing a task indicated by a guide video (601). The program may be configured to provide a user interface (e.g., UI screen, UI element, image, text, or multimedia) for each action in the set of actions.

[0288] According to one embodiment, the device information (621) of the electronic device (200) may include at least some of application information (e.g., information about version, category, type, application identifier, application UI, application function, etc.) for application(s) installed in the electronic device (200), software information (e.g., information about OS version, UI / UX version, etc.), or hardware information (e.g., information about device identifier, camera, biometric sensor, etc.).

[0289] In one embodiment, the MLLM model (614) may be a generative AI model trained to generate and / or provide a program (632). The MLLM model (614) may generate the program (632) based on text embedding data (624) for the description (631) and device information (621) of the electronic device (200). The MLLM model (614) may be a large-scale language model capable of processing various types of data, such as images and audio, in addition to text. The MLLM model (614) may understand the architecture of the text embedding data (624) corresponding to the description (631) and adjust the architecture according to the device information (621).

[0290] In one embodiment, the MLLM model (614) may generate a program (632) suitable for the electronic device (200). For example, the program (632) may include step-by-step (or execution order-based) UI screens for a set of multiple actions to be executed on the electronic device (200). For example, the program (632) may be for setting up multiple actions (unit / individual actions) to be automatically executed on the electronic device (200).

[0291] According to one embodiment, the program (632) may be generated in the form of code (e.g., a script) that can be directly executed without installation, or may be generated in the form of an application that requires installation (e.g., a mobile application package such as an '.apk' extension file for Android OS or an '.ipa' extension file for iOS). Depending on the runtime environment (e.g., JVM, Python Interpreter, etc.) existing within the electronic device (200), a conversion process such as code compilation may additionally be executed.

[0292] In some embodiments, an AI model that integrates at least some of the AI ​​models (611, 612, 613, 614) illustrated in FIG. 6 may be used, or only some of the AI ​​models (611, 612, 613, 614) may be selectively used.

[0293] According to one embodiment, the electronic device (200) of FIG. 2 may be configured to include at least some of the AI ​​models (611, 612, 613, 614) illustrated in FIG. 6. At least some of the AI ​​models (611, 612, 613, 614) may be included in the electronic device (200) (e.g., memory (220)) in the form of on-device AI models, but is not limited thereto. For example, at least some of the AI ​​models (611, 612, 613, 614) may be included in an AI server (e.g., server (108) of FIG. 1) and / or an electronic device of another user (e.g., electronic device (102, 104) of FIG. 1).

[0294] FIG. 7 is an example of a user interface related to a process of editing a description of an electronic device (200) according to one embodiment.

[0295] In one embodiment, the electronic device (200) can modify (or edit) the description. For example, the description may include task-specific data for at least one operation required to perform the task (e.g., text describing each operation).

[0296] According to one embodiment, the electronic device (200) can display a description of at least one operation used to generate the program and modify (or edit) the description based on user input.

[0297] Referring to FIG. 7, according to one embodiment, the electronic device (200) may display UI screens, such as a first screen (710), a second screen (720), a third screen (730), and a fourth screen (740), through the display (230). The first screen (710), the second screen (720), the third screen (730), and the fourth screen (740) may be for modifying (or editing) a description for at least one task.

[0298] According to one embodiment, the electronic device (200) may display a first screen (710). The first screen (710) may include an object (711) (e.g., an icon) corresponding to a program. The object (711) of FIG. 7 may correspond to the object (451) of FIG. 4.

[0299] According to one embodiment, the electronic device (200) may display, based on a user input (e.g., a touch on an icon) for an object (711) corresponding to a program, at least one of a UI element of a first option for executing the program (e.g., an execution menu of a first menu window (721)), a UI element of a second option for editing the program (or a description corresponding to the program, an action set by the program) (e.g., an edit menu of the first menu window (721)), or a UI element of a third option for sharing the program with another user (e.g., a share menu of the first menu window (721)).

[0300] For example, when a user input (e.g., touch) occurs on an object (711) within a first screen (710), the electronic device (200) may display a second screen (720). A first menu window (721) (or option window) regarding a simple action function may appear on the second screen (720). The first menu window (721) may include a run menu, an edit menu, and a share menu.

[0301] For example, when a user input (e.g., touch) for selecting the above-described modification menu within the first menu window (721) occurs, the electronic device (200) may display a third screen (730). A second menu window (731) may appear on the third screen (730). The second menu window (731) may display a description corresponding to specific actions set by the program (e.g., actions for issuing a family relationship certificate task). The second menu window (731) may be displayed together with an edit button (732).

[0302] According to one embodiment, a user can modify (or edit) a description using an edit button (732) within the third screen (730). For example, if the XX Bank app is not installed on the electronic device (200), the electronic device (200) can modify the action “launch XX Bank app” among the actions constituting the description to the action “launch YY Bank app” based on user input.

[0303] In one embodiment, the electronic device (200) may display a fourth screen (740). The fourth screen (740) may display a third menu window (741) indicating a modified description and a save button (742). Depending on the modified description, specific actions to be executed on the electronic device (200) may be displayed stepwise or sequentially. The electronic device (200) may store the modified description in response to a user input (e.g., a touch) to the save button (742) within the fourth screen (740).

[0304] In one embodiment, the electronic device (200) may update the program to correspond to the modified description in response to a user input (e.g., a touch) to a save button (742) within the fourth screen (740).

[0305] FIG. 8 is an example of a user interface related to a process in which an electronic device (200) shares a program according to one embodiment.

[0306] According to one embodiment, a user of an electronic device (200) can share a program (and / or a description corresponding to the program) with another user (e.g., a conversation partner) of an external electronic device (e.g., an electronic device (102, 104) of FIG. 1).

[0307] According to one embodiment, a conversation session may be established (e.g., a chatting state) between a user's electronic device (200) and another user's external electronic device, in which at least one message is exchanged. The electronic device (200) may share (send / receive) a program and / or a description corresponding to the program through the conversation session.

[0308] Referring to FIG. 8, the electronic device (200) can display UI screens, such as a first screen (810), a second screen (820), a third screen (830), and a fourth screen (840), through the display (230). The first screen (810), the second screen (820), the third screen (830), and the fourth screen (840) may be for sharing.

[0309] For example, the first screen (810) and the fourth screen (840) may be execution screens of applications (e.g., chat apps, messenger apps) running on the electronic device (200). The second screen (820) and the third screen (830) may be home screens of the electronic device (200).

[0310] For example, the conversation window (811) of the first screen (810) shows the conversation between the user of the electronic device (200) and another user.

[0311] According to one embodiment, while a user of an electronic device (200) is conversing with another user through a conversation window (811) (e.g., while a chat app is running), the screen may be switched from a first screen (810) to a second screen (820) based on user input.

[0312] According to one embodiment, the electronic device (200) may display a second screen (820) including an object (821) corresponding to a program. When a designated user input (e.g., a long touch) occurs for the object (821), the electronic device (200) may display a menu window (831) regarding a simple action function as illustrated in the third screen (830). The menu window (831) may include a share menu.

[0313] According to one embodiment, when a user input (e.g., touch) occurs on the shared menu within the menu window (831), the electronic device (200) may perform a sharing operation to transmit the program to another user's external electronic device using the application (e.g., chat app, messenger app). As the sharing operation is performed, the third screen (830) may automatically switch to the fourth screen (840).

[0314] According to one embodiment, the electronic device (200) may display a notification window (841) notifying the transmission of the program on the fourth screen (840). A screen corresponding to the fourth screen (840) including the notification window (841) may be displayed on an external electronic device. Another user of the external electronic device may select (e.g., touch) a UI element corresponding to the notification window (841) within the screen to download the program.

[0315] In this way, a program can be shared between a user of the electronic device (200) and another user of an external electronic device.

[0316] According to one embodiment, a program and / or a description corresponding to the program may be shared between a user of the electronic device (200) and another user of an external electronic device.

[0317] In one embodiment, another user's external electronic device can use the device information of the external electronic device to update the shared program to a version suitable for the external electronic device. In one embodiment, another user's external electronic device can use the shared description and the device information of the external electronic device to create a new program suitable for the external electronic device.

[0318] According to one embodiment, when sharing a program (or description), the electronic device (200) can identify whether the program (or description) contains personal information (or sensitive information). If the program (or description) contains personal information, the electronic device (200) can share a version of the program (or description) that excludes the personal information.

[0319] According to one embodiment, when generating a program, the electronic device (200) can identify whether the program (or the action set of the program) includes personal information (or sensitive information). If the program (or the action set of the program) includes the personal information, the electronic device (200) can display a UI element (e.g., a pop-up window) asking whether to agree to the use of the personal information through the display (230). If the electronic device (200) detects a user input agreeing to the use of the personal information through the UI element, the electronic device (200) can include the personal information in the program. If the electronic device (200) detects a user input rejecting the use of the personal information through the UI element, the electronic device (200) can exclude (or delete) the personal information from the program or perform anonymization processing (e.g., use restriction processing, display prohibition processing, sharing prohibition processing) on ​​the personal information.

[0320] FIG. 9 is an example of a user interface related to a process (e.g., operation 340 of FIG. 3) in which an electronic device (200) generates a program using text according to one embodiment.

[0321] According to one embodiment, the electronic device (200) can generate a program based on text. The electronic device (200) can obtain the program generated based on the text by inputting the text into at least one generative AI model. The program may be for performing a task (e.g., issuing a family relationship certificate). The program may be for setting and / or executing a set of actions related to the task (e.g., execution support, automatic execution, simple execution).

[0322] Referring to FIG. 9, the electronic device (200) can display UI screens, such as a first screen (910), a second screen (920), a third screen (930), a fourth screen (940), and a fifth screen (950), through a display (230). The first screen (910), the second screen (920), the third screen (930), the fourth screen (940), and the fifth screen (950) can be used to create a program using text.

[0323] The number of screens shown in Figure 9 is merely an example for illustrative purposes, and the number of screens displayed during the program creation process is not limited thereto. For example, additional screens may be created, or the number of screens may be reduced.

[0324] The first screen (410) and the second screen (420) are for explaining an operation of obtaining text corresponding to content indicating a task to be performed (e.g., operation 310 of FIG. 3).

[0325] In one embodiment, the text may be generated by an external electronic device (e.g., electronic device (102, 104) of FIG. 1) of another user and transmitted to the electronic device (200).

[0326] In one embodiment, the text may be received via a conversation session (e.g., a conversation session for a chat app) in which at least one message is exchanged between a user of the electronic device (200) and another user of an external electronic device (e.g., electronic device (102, 104) of FIG. 1 ).

[0327] According to one embodiment, the electronic device (200) can generate a program based on text (e.g., keywords, phrases, natural language sentences).

[0328] According to one embodiment, the electronic device (200) may display a first screen (910). The first screen (410) may display a conversation window (911) containing text. For example, the text in the conversation window (911) may include at least one of a keyword, a phrase, or a natural language sentence. For example, the text may be received from another user of the external electronic device through a conversation session (e.g., a conversation session for a chat app). For example, the text may include a description shared from another user of the external electronic device (e.g., the electronic devices (102, 104) of FIG. 1). For example, the description may correspond to a task (e.g., issuing a family relationship certificate) generated by the external electronic device. The description may describe at least one operation required to perform the task.

[0329] According to one embodiment, when a user input (e.g., a long touch) occurs in a dialogue window (911) within a first screen (910), the electronic device (200) may display a second screen (920). A menu window (921) regarding a simple action function may appear on the second screen (920). The menu window (921) may include an action setting menu. For example, the action setting menu may be for requesting the creation of a program.

[0330] According to one embodiment, the electronic device (200) may determine text (e.g., a description) within the conversation window (911) as content representing a task based on a user input (e.g., a long touch) to the conversation window (911) and / or a user input (e.g., a touch) selecting the action setting menu. The electronic device (200) may obtain (e.g., identify, extract, copy) the text.

[0331] The third screen (930) and the fourth screen (940) are for explaining the operation of generating a program based on the above text (e.g., operation 340 of FIG. 3).

[0332] According to one embodiment, the electronic device (200) may initiate (or trigger) the creation of a program in response to a user input (e.g., a touch) selecting the action setting menu within the menu window (921). While the program is being created, a third screen (930) may be displayed. The third screen (930) may include a notification window (931). The notification window (931) may display a message indicating that the program is being created (e.g., "Creating action...").

[0333] According to one embodiment, the electronic device (200) can obtain (e.g., extract from memory (220)) device information (e.g., at least one of application information, software information, or hardware information) of the electronic device (200).

[0334] According to one embodiment, the electronic device (200) may generate a program for performing a task, which is suitable for the electronic device (200), based on the text (e.g., description) in the dialogue window (911) and the device information of the electronic device (200). For example, the electronic device (200) may input text corresponding to a description received from an external electronic device and device information of the electronic device (200) into at least one generative AI model (e.g., MLLM model (614) of FIG. 6, MLLM model (1014) of FIG. 10), thereby obtaining a program generated based on the text and the device information.

[0335] In one embodiment, the program may be for performing a specific task (e.g., issuing a family relationship certificate). In relation to the task, the program may be for setting and / or executing at least one specific action (e.g., unit / individual actions for the family relationship certificate issuance task) to be executed on the electronic device (200).

[0336] According to one embodiment, the electronic device (200) may display a fourth screen (940) when the creation of the program is completed. The fourth screen (940) may include a notification window (941). The notification window (941) may display a message notifying the completion of the creation of the program.

[0337] The fifth screen (950) is intended to explain an operation (e.g., operation 350 of FIG. 3) for displaying a user interface related to the above program.

[0338] According to one embodiment, the electronic device (200) may display an object (951) (e.g., an icon) corresponding to a program through the fifth screen (950). The object (951) may be for performing a task. The object (951) may be for executing the program (or an action set corresponding to the program). For example, when a user input (e.g., a touch) occurs for the object (951), a task corresponding to the program may be performed. For example, when a user input (e.g., a touch) occurs for the object (951), at least one action set by the program corresponding to the object (951) may be executed (or triggered). For example, when a user input (e.g., a touch) occurs for the object (951), an action set (at least one action) set by the program corresponding to the object (951) may be automatically executed in batches.

[0339] FIG. 10 is a diagram illustrating a process in which an electronic device (200) according to one embodiment generates a program from text (e.g., guide text) using at least one generative AI model.

[0340] The LLM model (1013), MLLM model (1014), text embedding data (1041), text embedding data (1042), device information (1021), description (1031), and program (1032) (second program) of FIG. 10 may correspond to the LLM model (612), MLLM model (614), text embedding data (623), text embedding data (624), device information (621), description (631), and program (632) of FIG. 6, respectively.

[0341] In one embodiment, text included in the guide text (1023) and / or metadata of a shared program (1022) (first program) may be used as input to the action parsing AI model (1011). The metadata may be included in the program (1022) (first program) or received together with the program (1022) (first program).

[0342] For example, the electronic device (220) can generate a program (1032) (second program) using guide text (1023).

[0343] For example, the electronic device (200) can receive a shared program (1022) (first program) and / or metadata from an external electronic device. The electronic device (200) can regenerate a new program (1032) (second program) using the metadata of the program (1022) (first program).

[0344] According to one embodiment, the action parsing AI model (1011) can extract a text data set from guide text (1023) indicating a task to be performed.

[0345] For example, the action parsing AI model (1011) can filter out parts that are irrelevant to the task (e.g., unnecessary data such as names, trademarks, etc.) from the metadata and / or guide text (1023) of the program (1022) (first program).

[0346] According to one embodiment, the action parsing AI model (1011) can extract a text data set related to a task and output text embedding data (1041) that embeds the extracted text data set.

[0347] In one embodiment, text embedding data (1041) can be used as input to an LLM model (1013).

[0348] According to one embodiment, the LLM model (1013) can generate a description (1031) for at least one task using the text embedding data (1041). For example, the description (631) may correspond to a text data set that organizes multiple tasks required to perform a task in a step-by-step or sequential manner.

[0349] In one embodiment, the description (1031) and / or text embedding data (1042) for the description (1031) can be used as input to the MLLM model (1014).

[0350] According to one embodiment, device information (1021) of an electronic device (200) and / or text embedding data (1042) for the device information (1021) may be input into an MLLM model (1014).

[0351] According to one embodiment, the MLLM model (1014) can generate a program (1032) (second program) using text embedding data (1042) for the description (1031) and / or device information (1021).

[0352] FIG. 11 is an example of a user interface related to a process of setting a user routine using a simple action function of an electronic device (200) according to one embodiment.

[0353] Referring to FIG. 11, the electronic device (200) can display UI screens such as a first screen (1110), a second screen (1120), a third screen (1130), and a fourth screen (1140). The UI screens may be for adding a previously created program to a user routine.

[0354] According to one embodiment, the electronic device (200) can cause at least one action set by the program to be repeatedly executed by adding the program to the user routine.

[0355] According to one embodiment, the electronic device (200) may display an object (1111) corresponding to a previously generated program through the first screen (1110). When a user input (e.g., a long touch) is made to the object (1111), a routine setting menu (1112) may appear. The electronic device (200) may add the program to a user routine according to a user input of selecting (e.g., touching) the routine setting menu (1112).

[0356] According to one embodiment, the electronic device (200) may cause the task corresponding to the program to be repeatedly performed based on the routine execution condition by adding the program to a user routine (or setting it as a user routine) and / or storing the routine execution condition of the user routine in the memory (220).

[0357] According to one embodiment, the electronic device (200) can display a routine setting icon (1121) through a second screen (1120). When the electronic device (200) detects a user input (e.g., a touch) for selecting the routine setting icon (1121), the electronic device (200) can switch the second screen (1120) to a third screen (1130) for routine setting. When the electronic device (200) detects a user input (e.g., a touch) for selecting a routine menu (1131) within the third screen (1130), the electronic device (200) can switch the third screen (1130) to a fourth screen (1140). The user can use the routine addition button (1141) within the fourth screen (1140) to add at least one specific action (at least one unit / individual action) set by a program corresponding to the object (1111) to the user routine in bulk. At least one specific action added to a user routine can be executed repeatedly (e.g. periodically or whenever a specified condition is met).

[0358] FIG. 12 is an example of a user interface related to a process of setting a user routine and routine execution conditions using a simple action function of an electronic device (200) according to one embodiment.

[0359] The electronic device (200) can generate a program for a set of actions (at least one action) to be repeatedly executed in the step of setting (or adding) a user routine. In addition, the electronic device (200) can set routine execution conditions to be used as execution conditions for the program according to the user's intention.

[0360] Referring to FIG. 12, the electronic device (200) may display a first screen (1210). The first screen (1210) may include a first UI area (1211) for setting actions according to a user routine and a second UI area (1212) for setting routine execution conditions.

[0361] A user can set actions according to a user routine using the menus in the first UI area (1211). A user can set action execution conditions for at least some of the actions as routine execution conditions using the menus in the second UI area (1212).

[0362] The first menu window (1220) is an example of a UI element representing the first actions (first unit / individual actions) according to a user routine. The user can set the first action set (first actions, e.g., 1. Run YouT app (designated streaming app), 2. Search and watch the latest stock video, 3. Set Korean subtitles, 4. Set volume / screen brightness, 5. Exit YouT app (designated streaming app)) to be repeatedly executed (routineized) by using the menus (e.g., add action, set volume, whether to notify) of the first UI area (1211). The user can create a first program for the first action set and set the first action set as a user routine by selecting (e.g., touching) the action setting button (1221) in the first UI area (1211). Accordingly, specific actions of the first action set can be set and / or automatically executed as a user routine.

[0363] The first menu window (1220) may include an action setting button (1221). When a user selects (e.g., touches) the action setting button (1221), a first program for the first action set (first actions) displayed in the first menu window (1220) may be created, and an object (1230) corresponding to the first program may be displayed.

[0364] The second menu window (1260) is an example of a UI element representing a second set of actions (second actions) related to a task among user routines (e.g., a task of issuing a financial certificate). By selecting (e.g., touching) an action setting button (1261) within the second menu window (1260), the user can create a second program for the second set of actions of the task and set the second set of actions as a user routine. The third menu window (1250) is an example of a UI element for setting a routine execution condition. The user can set the execution condition of the program for the second set of actions using the action condition setting button (1251) within the third menu window (1250). For example, if either a first condition, in which a financial certificate login attempt occurs, or a second condition, in which a financial certificate expiration notification occurs, is satisfied, the second set of actions can be executed (or triggered).

[0365] According to one embodiment, the routine execution conditions may include not only the execution conditions of the program itself (e.g., location, time), but also conditions specific to each action of the program (conditions for each unit / individual action, e.g., in the case of a certificate issuance action, a condition for the expiration of the financial certificate validity period).

[0366] As illustrated in FIGS. 11 and 12, by using the simple action function to set user routines and / or routine execution conditions, it is possible to easily implement enhanced user routines (e.g., complex scenarios in which multiple applications interact, user preferences are adaptively reflected, or conditions for each action are different, customized / personalized scenarios) beyond simple routines (e.g., a routine that plays music at a specified time every morning).

[0367] FIG. 13 is an example of a user interface related to a process (e.g., operation 310 of FIG. 3) in which an electronic device (200) obtains content for program creation using external content according to one embodiment.

[0368] According to one embodiment, an electronic device (200) (e.g., a processor (210)) may extract at least a portion of external content and use the extracted content as content (e.g., guide content) for program generation. For example, the external content may be obtained from an external electronic device (e.g., an electronic device (102, 104) of FIG. 1, a server (108) of FIG. 1).

[0369] Referring to FIG. 13, the electronic device (200) can display a first screen (1310) for external content (e.g., a web page including various forms of content such as video, image, voice, and text).

[0370] According to one embodiment, the electronic device (200) can extract at least a portion of external content displayed on the first screen (1310) based on user input. For example, if the user presses the Get Action button (1311) and then selects a desired area, the image or audio within the selected area can be extracted as guide content. For example, if the user presses the Get Action button (1311) and then designates a portion of an area through a drag input, the text within the corresponding area can be extracted as guide content.

[0371] According to one embodiment, the electronic device (200) may obtain a first description (1320) for a plurality of tasks required to perform a task (e.g., a task of issuing a financial certificate) using guide content (e.g., a video, an image, an audio, a text, a multimedia) extracted from the first screen (1310). For example, the first description (1320) may set at least one task (e.g., a common action of banking apps) required to perform the task. The electronic device (200) may convert the first description (1320) into a second description (1330) according to the device information of the electronic device (200). The second description (1330) may set an action set (at least one action) processed to be suitable for the electronic device (1310). For example, the action set of the second description (1330) may correspond to at least one task of the first description (1320). For example, the action set of the second description (1330) may include actions that modify, transform, replace, update, or specialize the tasks of the first description (1320). For example, the action set of the second description (1330) may include actions added in relation to the tasks of the first description (1320).

[0372] According to one embodiment, the electronic device (200) may generate a program that sets the action set (e.g., a plurality of actions) required to perform the task (e.g., a task of issuing a financial certificate) on the electronic device (200) using the second description (1330). As the program is executed, the electronic device (200) may execute an action of the action set by the program or display a second screen (1340), which is a UI screen related to the action.

[0373] FIG. 14 is an example of a process in which an electronic device (200) according to one embodiment provides a simple action function using an intelligent assistant.

[0374] According to one embodiment, the intelligent assistant may be a module that interacts with a user of the electronic device (200) via voice and / or text. For example, the intelligent assistant may be implemented in the form of an application (or software) installed on the electronic device (200).

[0375] According to one embodiment, in operation 1401, the electronic device (200) may receive a user request (1411) (e.g., voice, text) through the first UI screen (1410) of the intelligent assistant. For example, the user may input a voice query, such as "How do I issue a financial certificate in a banking app?", through the user interface of the intelligent assistant. For example, the voice query may be converted into text and displayed on the first UI screen (1410).

[0376] According to one embodiment, in operation 1402, the electronic device (200) may determine, based on a user request (1411) received through an intelligent assistant (or a user interface of the intelligent assistant), at least one of a target application (1421) (e.g., XX Bank app), a specific function (e.g., a unit function of the target application (1421)) that can be provided by the electronic device (200) in relation to the target application (1421), or a specific target function (e.g., a function provided at the framework level, a common function provided by the electronic device (200) that is not an application level, such as a virtual keyboard or an AI panel) that can be provided by the electronic device (200).

[0377] According to one embodiment, the electronic device (200) may determine the target application (1421) by additionally considering the user's application usage history. For example, if multiple applications (e.g., XX Bank app, YY Bank app) related to a user request (1411) are installed on the electronic device (200), an application with a high frequency of use or recently installed may be determined as the target application (1421).

[0378] According to one embodiment, the electronic device (200) may obtain a first description (1422) by inputting text corresponding to a user request (1411) into at least one generative AI model (e.g., the LLM model (613) of FIG. 6 and the LLM model (1013) of FIG. 10). For example, the first description (1422) may be a guide text indicating a task to be performed (e.g., a method for issuing a financial certificate in a banking app).

[0379] In one embodiment, the first description (1422) may include at least one operation required to perform a task (e.g., a method for issuing a financial certificate in a banking app) according to a user request (1411). For example, the first description (1422) may include multiple operations constituting a method for issuing a financial certificate in a banking app.

[0380] In one embodiment, the first description (1422) may be identical to, but not limited to, a description generated by at least one generative AI model. For example, the first description (1422) may be a description in which some of the tasks included in the description are deleted, replaced, or new tasks are added.

[0381] In one embodiment, the first description (1422) may be configured as a set of tasks for general-purpose tasks that may be commonly (or universally) used across various applications (e.g., all banking apps, general-purpose banking apps).

[0382] According to one embodiment, in operation 1403, the electronic device (200) may generate a program for performing a task based on the first description (1422). The program may be for setting an action set (at least one action) for performing a task (e.g., issuing a financial certificate in a banking app). For example, at least one action included in the action set of the program may correspond to at least one task included in the first description (1422).

[0383] According to one embodiment, the electronic device (200) may generate a program for a set of actions to be executed on the electronic device (200) based on text and application information (e.g., information about the target application (1421) and / or the specific function) corresponding to the first description (1422).

[0384] According to one embodiment, the electronic device (200) can obtain a program generated by at least one generative AI model by inputting text corresponding to a user request (1411) and application information (e.g., information about the manufacturer, version, category, type, application identifier, application UI, application function, etc.) into at least one generative AI model (e.g., MLLM (614) of FIG. 6, MLLM (1014) of FIG. 10).

[0385] According to one embodiment, the program may include a set of actions to be executed on the electronic device (200) to perform a task (e.g., issuing a financial certificate). The program may further include UI information (e.g., object information, UI element information, UI screen information for each action) for performing the task. For example, the program may provide UI information for a third UI screen (1440) and guide UI elements (1441, 1442) to be displayed when executed. For example, the UI information included in the program may be generated (or provided) by at least one generative AI model (e.g., the MLLM model (614) of FIG. 6, the MLLM model (1014) of FIG. 10).

[0386] According to one embodiment, the program may include UI information regarding a third screen (530) related to the first action, a fourth screen (540) related to the second action, and a fifth screen (550) related to the third action. The electronic device (200) may display a user interface related to the program based on the UI information included in the program (e.g., UI screen information for each action).

[0387] According to one embodiment, the electronic device (200) may display a user interface related to the program through the user interface of the intelligent assistant. For example, the electronic device (200) may display a second UI screen (1430) including an object (1431) corresponding to the program (e.g., a button for issuing a financial certificate) and a set of actions (1432) to be executed, based on UI information included in the program.

[0388] According to one embodiment, in operation 1404, the electronic device (200) may execute a set of actions (at least one action) included in the program step by step or sequentially. The electronic device (200) may display a third UI screen (1440) in response to a user input (e.g., a touch) to an object (1431) within the second UI screen (1430). The third UI screen (1440) may be an execution screen of a target application (1421) (e.g., XX Bank app). In one embodiment, the electronic device (200) may display guide UI elements (1441, 1442) (e.g., color / highlight box, guide message such as “Please log in / select authentication menu”) for parts requiring user input within the third UI screen (1440) based on UI information included in the program.

[0389] FIG. 15 is another example of a process in which an electronic device (200) according to one embodiment provides a simple action function using an intelligent assistant.

[0390] Action 1501, the first UI screen (1510) and the user request (1511) may correspond to actions 1401, the first UI screen (1410) and the user request (1411) of FIG. 4, respectively.

[0391] In operation 1502, the electronic device (200) may obtain a first description (1522) by inputting a guide text according to a user request (1511) into a first AI model (e.g., the LLM model (613) of FIG. 6). The first description (1522) may be composed of a set of tasks (at least one task) that can be commonly (or universally) used in various applications (e.g., all banking apps, universal banking apps).

[0392] In operation 1503, the electronic device (200) may obtain a second description (1530) by inputting UI information of a first description (1522) and a target application (1521) (e.g., XX Bank app) into a second AI model (e.g., MLLM model (614) of FIG. 6). The second description (1530) may include an action set (at least one action) specialized for the target application (e.g., XX Bank app). For example, the action set of the second description (1530) may correspond to the task set of the first description (1522). For example, the action set of the second description (1530) may be obtained by deleting, replacing, or adding new tasks to some of the tasks included in the task set of the first description (1522).

[0393] In operation 1504, the electronic device (200) may generate a program for a set of actions to be executed on the electronic device (200) using the second description (1530). The electronic device (200) may display a second UI screen (1540) including an object (1541) corresponding to the program (e.g., a button for issuing a financial certificate) and the second description (1530).

[0394] Action 1505 and the third UI screen (1550) may correspond to action 1404 and the third UI screen (1440), respectively.

[0395] FIG. 16 is another example of a process in which an electronic device (200) according to one embodiment provides a simple action function using an intelligent assistant.

[0396] Action 1601 and the first UI screen (1610) may correspond to action 1401 and the first UI screen (1410) of FIG. 4, respectively. The first UI screen (1610) may be a user interface of an intelligent assistant.

[0397] Action 1602, target application (1621) and description (1622) may correspond to action 1402, target application (1421) and first description (1422), respectively.

[0398] If the target application (1621) (e.g., a banking app) is a secure application or an application that requires authentication due to high personal information sensitivity (or security level), the electronic device (200) may not provide UI information of the target application (1621) to an external AI model.

[0399] In such a case, the electronic device (200) can generate a program using only a description (1622) for general-purpose tasks that can be commonly (or universally) used in various applications.

[0400] In operation 1603, the electronic device (200) may display an object corresponding to the program (e.g., a button for issuing a financial certificate) through a user interface of the intelligent assistant (e.g., a second UI screen (1630)).

[0401] Action 1604 and the third UI screen (1640) may correspond to actions 1404 and the third UI screen (1440), respectively. The third UI screen (1640) may be an execution screen of a target application (1621) (e.g., XX Bank app).

[0402] In one embodiment, during the process of generating a program for performing a task, it may be impossible to verify UI information of a target application (1621) related to the task. For example, the target application (1621) may be secured or have a relatively high security level.

[0403] In this case, an external AI model may not be permitted to access the target application (1621). The UI information of the target application (1621) may not be provided to the AI ​​model. Consequently, analysis of the UI information of the target application (1621) by the AI ​​model may be impossible. The UI information of the target application (1621) may not be reflected in the program generated by the AI ​​model.

[0404] According to one embodiment, the electronic device (200) may execute (e.g., call) an internal application manager to execute a program, and execute a target application (1621) through the application manager. The application manager may execute a program generated based on a description (1622) of general tasks, thereby executing the target application (1621) and / or functions of the target application (1621). The application manager may analyze UI information within the UI screen (1640) of the target application (1621) in real time while executing the target application (1621). The manager may additionally display or update UI elements in real time based on the UI information analysis result. For example, an action requiring user authentication or input of personal information (or a specific function of the target application (1621)) may be executed while executing the program. In this case, the application manager may identify an area requiring user authentication or input of personal information (e.g., an input window) based on the UI information of the target application (1621). The above application manager may display guide UI elements (e.g., colored / highlighted boxes around input windows, guide messages such as "Log in / Select an authentication menu") around the corresponding area. In one embodiment, data of the UI screen (1640) displayed on the screen may be input to an AI model for UI analysis to obtain UI information (e.g., UI component information, information on the functions performed by each UI component, etc.).

[0405] According to one embodiment, the UI screen displayed on the screen may be a screen generated using an AI model trained to configure and generate a UI to be displayed on the screen. For example, the electronic device (200) may display a UI screen generated by applying guide UI elements by transmitting guide UI element information as input to an AI model for UI generation based on a description (1622) included in the generated program.

[0406] FIG. 17 is another example of a process in which an electronic device (200) according to one embodiment provides a simple action function using an intelligent assistant.

[0407] In one embodiment, a target application (e.g., Samsung Wallet app) of the electronic device (200) may be executed in the background. UI screens (1710, 1720, 1730, 1740) for simple action functions may be provided through the user interface of the intelligent assistant of the electronic device (200).

[0408] In operation 1701, the electronic device (200) may receive a user request (1711) (e.g., voice, text) via a first UI screen (1710), which is a user interface of an intelligent assistant. For example, the user may input a voice query, such as "How do I add a card to Samsung Wallet?", through the intelligent assistant in relation to a task to be performed (e.g., a task of adding a card to Samsung Wallet). The voice query may be converted into text and displayed on the first UI screen (1710).

[0409] In operation 1702, the electronic device (200) may input guide text and UI information of a target application (e.g., Samsung Wallet app) according to a user request (1711) into an AI model (e.g., LLM model (613) of FIG. 6). In response to the input, the electronic device (200) may obtain a description (1721) for at least one task. The description (1721) may represent an action set for a plurality of step-by-step (or sequential) actions required to perform a task (e.g., adding a card to Samsung Wallet) according to the user request (1711).

[0410] In operation 1702, the electronic device (200) can generate a program for the action set using the description (1721). The electronic device (200) can display a second UI screen (1720) including an object (1722) corresponding to the generated program (e.g., a button to add a card to Samsung Wallet) and the description (1721) through the user interface of the intelligent agent.

[0411] In operation 1703, the electronic device (200) may execute a target application (e.g., a Samsung Wallet app) to perform a task in the background in response to a user input (e.g., a touch) on the object (1722). The electronic device (200) may use the target application to execute a plurality of actions set by the program. While the actions are being executed, actions requiring user input may occur.

[0412] In steps 1703 and 1704, the electronic device (200) may extract UI elements (e.g., a next execution button (1731), a wallet app execution button (1741)) for receiving user input from a target application (e.g., a Samsung Wallet app). The next execution button (1731) may be a UI element for transitioning to the next screen. The wallet app execution button (1741) may be a UI element for executing the wallet app, which is a target application.

[0413] The electronic device (200) can display the extracted UI elements through the third UI screen (1730) and the fourth UI screen (1740), which are user interfaces of the intelligent assistant.

[0414] According to this, user convenience can be improved by providing simple action functions without cumbersome screen switching between different applications.

[0415] FIG. 18 is a diagram illustrating a system including a generative artificial intelligence model according to one embodiment.

[0416] Referring to FIG. 18, the User Query / Response Interface (1810) can receive a user's input. The user's input may be in the form of natural language, images, and / or videos. Furthermore, context information may also be transmitted when the user's input is transmitted. Context information may include various additional information at the time of user input. For example, information about the application the user is currently using or the user's location information. Furthermore, the user's input may be in a mixed form of the aforementioned natural language, images, sounds, and context information. Furthermore, the user's input may also be in a non-natural language form, such as selecting a menu. The User Query / Response Interface (1810) can output the results of a generative artificial intelligence system to the user. The output may be in the form of natural language or specific content, and may also be provided in the form of an action requested by the user. The User Query Interface can output the results of a generative artificial intelligence system to the user. The output may be in the form of natural language or specific content, and may also be provided in the form of an action requested by the user.

[0417] The AI ​​framework (1820) can receive user input and coordinate and control each component necessary to perform the user's intention based on the user's query.

[0418] User input received from the User Query / Response Interface (1810) can be transmitted to the Prompt design component (1821). The Prompt design component (1821) can be used to generate prompts suitable for inputting the user input into a Large Language Model (LLM) or a Large Multimodal Model (LMM). The Prompt design component (1821) can be an AI component that uses a machine learning algorithm or a neural network to develop better prompts over time. The Prompt design component (1821) can access a knowledge component (e.g., knowledge repositories (1840)) containing user preference data, a prompt library, and prompt examples based on the user input to generate a prompt, and transmit the generated prompt to the LLM or LMM.

[0419] The API / Plug-in management component (1823) can communicate with external information when there is a request for additional information when passing user input as input to a generative model. The API / Plug-in management component (1823) can establish a channel for communication with the outside of the AI ​​Interface through the API, and can enable access to various data sources (e.g., knowledge repositories (1840)) through the established channel. In addition, if the API / Plug-in management component (1823) needs to execute an action that ultimately executes the user input, rather than an intermediate result, in an application or service, it can request the action to the application / service component (1830) through the API. Information obtained from an external source can be used to generate a prompt in the Prompt design component (1821) together with the user input, or can be passed as input to the generative model.

[0420] The Refiner component (e.g., the output modification component (1825)) can fine-tune the output from a generative model. For example, the Refiner component can verify that the content generated by the LLM and / or LMM is not irrelevant, biased, or harmful. Furthermore, the Refiner component can determine the degree to which the output matches the user's desired result and, if necessary, perform additional processing. The Refiner component can also configure and provide users with hints to avoid undesirable output.

[0421] Generative AI Model (1850) can generally refer to an artificial intelligence neural network that creates new types of data based on user input information. Generative AI Model (1850) can include an image-generating model and / or a language-generating model. Representative image-generating models include a generative adversarial network (GAN) and a variational autoencoder (VAE), and examples include a diffusion-based generative model that uses a VAE and a transformer structure. A language-generating model is a model trained to statistically output the most appropriate output value based on input values, and representative examples include models such as CHAT-GPT 3 and CHAT-GPT 4. There are also LMMs that can recognize various types of data input, such as text, images, and voice, and generate new data corresponding to them.

[0422] According to one embodiment, the electronic device (101) of FIG. 1 and / or the electronic device (200) of FIG. 2 may be configured to include at least a portion of the User Query / Response Interface (1810), the AI ​​framework (1820), the application / service component (1830), the knowledge repositories (1840), or the Generative AI Model (1850) of FIG. 18. According to one embodiment, at least a portion of the User Query / Response Interface (1810), the AI ​​framework (1820), the application / service component (1830), the knowledge repositories (1840), or the Generative AI Model (1850) of FIG. 18 may be included in another electronic device (e.g., another user's electronic device (e.g., the electronic device (102, 104) of FIG. 1) and / or a server (e.g., the server (108) of FIG. 1).

[0423] An electronic device (e.g., electronic device (101) of FIG. 1, electronic device (200) of FIG. 2) according to one embodiment of the present disclosure may include a display (e.g., display module (160) of FIG. 1, display (230) of FIG. 2), at least one processor (e.g., processor (120) of FIG. 1, processor (210) of FIG. 2), and a memory (e.g., memory (130) of FIG. 1, memory (220) of FIG. 2)) that stores instructions. The instructions, when executed by the at least one processor, may cause the electronic device to obtain content indicating a task to be performed, obtain a description of at least one work required to perform the task based on the content, extract device information of the electronic device, generate a program for a set of actions to be executed in the electronic device based on the description and the device information, and display a user interface related to the program through the display.

[0424] According to one embodiment of the present disclosure, the instructions, when executed by the at least one processor, may cause the electronic device to perform at least one of the following operations: obtaining an image data set generated by a first artificial intelligence model by inputting a video corresponding to the content into the first artificial intelligence model; obtaining the description generated by the second artificial intelligence model by inputting the image data set into the second artificial intelligence model; and obtaining the program generated by the third artificial intelligence model by inputting at least a portion of the description and the device information into the third artificial intelligence model.

[0425] According to one embodiment of the present disclosure, the content may correspond to at least one of content selected based on a user input of the electronic device, content received through a conversation session in which at least one message is exchanged between a user of the electronic device and another user of an external electronic device, or content received through an intelligent assistant interacting with the user of the electronic device through voice.

[0426] According to one embodiment of the present disclosure, a user interface associated with the program may include an object corresponding to the program. The program may be executed based on a first user input to the object.

[0427] According to one embodiment of the present disclosure, based on a second user input for the object, a user interface element for at least one of a first option for editing the program or a second option for sharing the program with another user may be displayed through the display.

[0428] According to one embodiment of the present disclosure, the instructions, when executed by the at least one processor, may cause the electronic device to transmit at least one of the program or the description to an external electronic device through a communication circuit (e.g., the communication module (190) of FIG. 1, the communication circuit (240) of FIG. 2) based on a user input, receive a notification about the execution status of the task from the external electronic device through the communication circuit, and display the notification through the display.

[0429] According to one embodiment of the present disclosure, the instructions, when executed by the at least one processor, may cause the electronic device to store information on an execution condition of the program in the memory and execute the program based on satisfaction of the execution condition.

[0430] According to one embodiment of the present disclosure, the instructions, when executed by the at least one processor, may cause the electronic device to store information on a performance deadline and a notification condition of the task in the memory, and, based on the notification condition being satisfied, display a first notification notifying of the expiration of the performance deadline or a second notification guiding that execution of the program is required before the expiration of the performance deadline through the display.

[0431] According to one embodiment of the present disclosure, the at least one task may include a first task and a second task. The set of actions may include a first action provided through a first application and a second action provided through a second application. The first task may be coupled to the first action, which is executed through interaction with a user of the electronic device. The second task may be coupled to the second action, which is executed without interaction with the user.

[0432] According to one embodiment of the present disclosure, the instructions, when executed by the at least one processor, may cause the electronic device to highlight a user interface element of the first application corresponding to the first action as the program is executed, and to execute the first action based on a user input to the user interface element.

[0433] According to one embodiment of the present disclosure, at least one action from the set of actions may be executed by an external electronic device connected to the electronic device via a designated network. A notification regarding the execution status of the task may be displayed on the external electronic device.

[0434] According to one embodiment of the present disclosure, the instructions, when executed by the at least one processor, may cause the electronic device to display the description through the display, modify the description based on a user input, and update the program based on the modified description.

[0435] According to one embodiment of the present disclosure, the instructions, when executed by the at least one processor, may cause the electronic device to generate the program further based on personal information of a user of the electronic device.

[0436] According to one embodiment of the present disclosure, the instructions, when executed by the at least one processor, may cause the electronic device to repeatedly perform the task corresponding to the program based on the routine execution condition by adding the program to a user routine (or setting it as a user routine) and / or storing a routine execution condition of the user routine in the memory.

[0437] According to one embodiment of the present disclosure, the instructions, when executed by the at least one processor, may cause the electronic device to receive a user request using an intelligent assistant, determine a target application from among a plurality of applications installed on the electronic device based on the user request, and display a user interface related to the program using the intelligent assistant and the target application.

[0438] A method of operating an electronic device according to one embodiment of the present disclosure may include an operation of obtaining content indicating a task to be performed, an operation of obtaining a description of at least one work required to perform the task based on the content, an operation of extracting device information of the electronic device, an operation of generating a program for a set of actions to be executed in the electronic device based on the description and the device information, and an operation of displaying a user interface related to the program through a display of the electronic device.

[0439] According to one embodiment of the present disclosure, the operation of generating the program may include at least one of an operation of obtaining an image data set generated by a first artificial intelligence model by inputting a video corresponding to the content into a first artificial intelligence model, an operation of obtaining the description generated by the second artificial intelligence model by inputting the image data set into a second artificial intelligence model, and an operation of obtaining the program generated by a third artificial intelligence model by inputting at least a part of the description and the device information into a third artificial intelligence model.

[0440] According to one embodiment of the present disclosure, the content may correspond to at least one of content selected based on a user input of the electronic device, content received through a conversation session in which at least one message is exchanged between a user of the electronic device and another user of an external electronic device, or content received through an intelligent assistant interacting with the user of the electronic device through voice.

[0441] According to one embodiment of the present disclosure, a user interface associated with the program may include an object corresponding to the program. The method may further include an operation of executing the program based on a first user input for the object.

[0442] According to one embodiment of the present disclosure, the method may further include displaying, through the display, a user interface element for at least one of a first option for editing the program or a second option for sharing the program with another user, based on a second user input for the object.

[0443] According to one embodiment of the present disclosure, the method may further include an operation of transmitting at least one of the program or the description to an external electronic device based on a user input, an operation of receiving a notification about the performance status of the task from the external electronic device, and an operation of displaying the notification through the display.

[0444] According to one embodiment of the present disclosure, the method may further include an operation of storing information on an execution condition of the program, and an operation of executing the program based on satisfaction of the execution condition.

[0445] According to one embodiment of the present disclosure, the method may further include an operation of storing information on a performance deadline and a notification condition of the task, and an operation of displaying, through the display, a first notification notifying of an expiration of the performance deadline or a second notification guiding that execution of the program is required before the expiration of the performance deadline, based on the notification condition being satisfied.

[0446] According to one embodiment of the present disclosure, the method may further include an operation of displaying the description through the display, an operation of modifying the description based on a user input, and an operation of updating the program based on the modified description.

[0447] According to one embodiment of the present disclosure, the program may be generated further based on personal information of a user of the electronic device.

[0448] According to one embodiment of the present disclosure, the method may further include an operation of adding the program to a user routine (or setting it as a user routine) and / or an operation of storing a routine execution condition of the user routine in a memory of the electronic device. Based on the routine execution condition, the task corresponding to the program may be repeatedly performed.

[0449] According to one embodiment of the present disclosure, the method may further include an operation of receiving a user request using an intelligent assistant, an operation of determining a target application from among a plurality of applications installed on the electronic device based on the user request, and an operation of displaying a user interface related to the program using the intelligent assistant and the target application.

[0450] A storage medium according to one embodiment of the present disclosure may be a non-transitory computer-readable storage medium. The storage medium may record a program for executing a method of operating an electronic device. The storage medium may record a program for executing a method including an operation of obtaining guide content indicating a task to be performed, an operation of obtaining an operation sequence description for operations required to perform the task based on the content, an operation of extracting device information of the electronic device, an operation of generating a program for a set of actions to be performed in the electronic device based on the description and the device information, and an operation of displaying a user interface related to the program through a display of the electronic device.

[0451] In the present disclosure, a function or operation performed by an electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (200) of FIG. 2) may be performed by one or more processors executing one or more instructions stored in a memory. The function or operation of the electronic device mentioned in the present disclosure may be performed by one processor executing one or more instructions, or may be performed by a combination of multiple processors executing one or more instructions. The processor mentioned in the present disclosure may be understood to include a circuit for performing an operation or controlling other components of the electronic device. For example, the one or more processors may include a central processing unit (CPU), a microprocessor unit (MPU), an application processor (AP), a communication processor (CP), a neural processing unit (NPU), a system on a chip (SoC), or an integrated circuit (integrated circuit, IC) configured to execute one or more instructions. The one or more processors may be configured to perform the operation of the electronic device described above.

[0452] In the present disclosure, a program (software module, software) may be stored in a non-volatile memory including a random access memory (RAM), a flash memory, a read only memory (ROM), an electrically erasable programmable read only memory (EEPROM), a magnetic disc storage device, a compact disc ROM (CD-ROM), digital versatile discs (DVDs) or other forms of optical storage devices, a magnetic cassette. Or, it may be stored in a memory formed by a combination of some or all of these. The memory may be formed by a single storage medium, or may be formed by a combination of a plurality of storage media. The one or more commands may be stored in a single storage medium, or may be distributed and stored in a plurality of storage media.

[0453] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

[0454] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0455] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0456] Various embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0457] According to one embodiment, the method according to various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play Store™) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0458] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more actions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In electronic devices, display; at least one processor; and When executed by at least one processor, the electronic device: Obtain content indicating the task to be performed, Obtain a description of at least one work required to perform the task based on the above content, Extract device information of the above electronic device, Based on the above description and the above device information, generate a program for a set of actions to be executed on the electronic device, An electronic device comprising a memory storing instructions for displaying a user interface related to the program through the display.

2. In claim 1, The above instructions, when executed by the at least one processor, cause the electronic device to: An operation of obtaining an image data set generated by the first artificial intelligence model by inputting a video corresponding to the above content into the first artificial intelligence model; An operation of obtaining the description generated by the second artificial intelligence model by inputting the image data set into the second artificial intelligence model, and An electronic device that performs at least one of the operations of obtaining the program generated by the third artificial intelligence model by inputting at least a part of the description and the device information into the third artificial intelligence model.

3. In claim 1, The above content is, Content selected based on user input of the electronic device; Content received through a conversation session in which at least one message is exchanged between a user of said electronic device and another user of an external electronic device, or An electronic device, wherein at least one of the contents is received through an intelligent assistant that interacts with a user of the electronic device through voice.

4. In claim 1, The user interface associated with the above program includes an object corresponding to the above program, An electronic device in which the program is executed based on a first user input for the object.

5. In claim 4, An electronic device, wherein, based on a second user input for the object, a user interface element for at least one of a first option for editing the program or a second option for sharing the program with another user is displayed through the display.

6. In claim 1, The above instructions, when executed by the at least one processor, cause the electronic device to: transmitting at least one of the program or the description to an external electronic device via a communication circuit based on user input; Receive notification of the execution status of the task from the external electronic device through the communication circuit; An electronic device that displays the above notification through the above display.

7. In claim 1, The above instructions, when executed by the at least one processor, cause the electronic device to: Store information about the execution conditions of the above program in the above memory, An electronic device that executes the program based on the satisfaction of the above execution conditions.

8. In claim 1, The above instructions, when executed by the at least one processor, cause the electronic device to: Store information about the execution deadline and notification conditions of the above task in the above memory, An electronic device that displays, through the display, a first notification notifying the expiration of the execution deadline or a second notification guiding the necessity of executing the program before the expiration of the execution deadline, based on the satisfaction of the above notification condition.

9. In claim 1, wherein at least one of the above operations comprises a first operation and a second operation, The above action set includes a first action provided through a first application and a second action provided through a second application, An electronic device wherein the first task is coupled to the first action that is executed through interaction with a user of the electronic device, and the second task is coupled to the second action that is executed without interaction with the user.

10. In claim 9, The above instructions, when executed by the at least one processor, cause the electronic device to: As the above program is executed, highlighting a user interface element of the first application corresponding to the first action; An electronic device that executes the first action based on user input to the user interface element.

11. In claim 1, At least one action of the above action set is executed via an external electronic device connected to the electronic device via a designated network, An electronic device that displays a notification regarding the performance status of the above task on the external electronic device.

12. In the method of operating an electronic device, The act of obtaining content indicating the task to be performed; An action of obtaining a description of at least one work required to perform the task based on the above content; An operation of extracting device information of the electronic device; An operation of generating a program for a set of actions to be executed on the electronic device based on the description and the device information; and A method comprising the action of displaying a user interface related to the program through a display of the electronic device.

13. In claim 12, The action of generating the above program is: An operation of obtaining an image data set generated by the first artificial intelligence model by inputting a video corresponding to the above content into the first artificial intelligence model; An operation of obtaining the description generated by the second artificial intelligence model by inputting the image data set into the second artificial intelligence model; and A method comprising at least one of the actions of obtaining the program generated by the third artificial intelligence model by inputting at least a portion of the description and the device information into the third artificial intelligence model.

14. In claim 12, The above content is, Content selected based on user input of the electronic device; Content received through a conversation session in which at least one message is exchanged between a user of said electronic device and another user of an external electronic device, or A method for receiving at least one piece of content through an intelligent assistant interacting with a user of the electronic device via voice.

15. In claim 12, The user interface associated with the above program includes an object corresponding to the above program, A method further comprising executing the program based on a first user input for the object.

Images