Method for generating user interface by using generative artificial intelligence and electronic device therefor

Abstract

Disclosed is an electronic device comprising a display, a processor, and a memory. The electronic device may generate at least one prompt for generating a recommended user interface by using an artificial intelligence model, and generate a recommended UI by inputting the at least one prompt to a generative artificial intelligence model.

Description

Method for creating a user interface using generative artificial intelligence and electronic device therefor

[0001] The embodiments disclosed in this document relate to a method for generating a user interface using generative artificial intelligence and an electronic device therefor.

[0002] Recommendation services provided through electronic devices are becoming widespread. Electronic devices can recommend applications to users based on their usage history. For example, an electronic device can provide recommended search terms based on previous input history. Electronic devices can also recommend launching an application different from the currently running application. For example, if a received message contains address information, the electronic device may recommend launching a map application to search for a location corresponding to the address information. The user can exit the running message application and then search for an icon to launch the map application. After launching the map application, the user can search for the address and then find the recommended location.

[0003] The above information may be provided as background information to aid in understanding the present disclosure. No claim or determination is made as to whether any of the above is applicable as prior art in connection with the present disclosure.

[0004] An electronic device according to an embodiment disclosed in the present document may include a display, at least one processor electrically connected to the display, and a memory electrically connected to the at least one processor and storing instructions. The instructions, when executed by the at least one processor, may cause the electronic device to identify at least one of one or more keywords or one or more intentions from data including at least one of a message, schedule information, or input information using a first artificial intelligence model. The instructions, when executed by the at least one processor, may cause the electronic device to identify a recommended action based on at least one of the one or more keywords or the one or more intentions, and to generate at least one prompt for generating a recommendation UI using a second artificial intelligence model. The instructions, when executed by the at least one processor, may cause the electronic device to obtain the recommendation UI including guide information for the recommended action by inputting the at least one prompt into a third generative artificial intelligence model. The above instructions, when executed by the at least one processor, may cause the electronic device to display the recommendation UI on the display.

[0005] A method for providing a recommendation UI of an electronic device according to an embodiment disclosed in the present document may include an operation of identifying at least one of one or more keywords or one or more intentions from data including at least one of message information, schedule information, or input information using a first artificial intelligence model, an operation of identifying a recommendation action based on at least one of the one or more keywords or the one or more intentions, an operation of generating at least one prompt for generating a recommendation UI using a second artificial intelligence model, an operation of obtaining the recommendation UI including guide information for the recommendation action by inputting the at least one prompt into a third generative artificial intelligence model, and an operation of displaying the recommendation UI.

[0006] A computer-readable storage medium according to an embodiment disclosed in this document can store instructions that, when executed by a processor of an electronic device, cause the electronic device to perform the method for providing the recommended UI.

[0007] Figure 1 illustrates a communication environment of an electronic device according to one embodiment.

[0008] Figure 2 illustrates an artificial intelligence system according to one embodiment.

[0009] Figure 3 illustrates modules of an electronic device according to one embodiment.

[0010] FIG. 4 illustrates a data management module of an electronic device according to one embodiment.

[0011] FIG. 5 illustrates a message screen of an electronic device according to one embodiment.

[0012] FIG. 6 illustrates a trigger management module of an electronic device according to one embodiment.

[0013] FIG. 7 illustrates a UI generation module of an electronic device according to one embodiment.

[0014] Figure 8 is a flowchart of a method for generating a prompt in an electronic device according to one embodiment.

[0015] FIG. 9a illustrates a recommendation UI of an electronic device according to one embodiment.

[0016] FIG. 9b illustrates a recommendation UI of an electronic device according to one embodiment.

[0017] FIG. 9c illustrates a recommendation UI of an electronic device according to one embodiment.

[0018] FIG. 9d illustrates a recommendation UI of an electronic device according to one embodiment.

[0019] FIG. 10 illustrates additional information UIs of an electronic device according to one embodiment.

[0020] FIG. 11 illustrates a recommendation UI of an electronic device according to one embodiment.

[0021] Fig. 12 is a flowchart of a method for providing a UI of an electronic device according to one embodiment.

[0022] Fig. 13 is a flowchart of a method for obtaining a recommendation UI of an electronic device according to one embodiment.

[0023] FIG. 14 is a block diagram of an electronic device within a network environment according to various embodiments.

[0024] In connection with the description of the drawings, the same or similar reference numerals may be used for identical or similar components.

[0025] Hereinafter, various embodiments of the present invention will be described with reference to the attached drawings. However, this is not intended to limit the present invention to specific embodiments, and it should be understood that the present invention encompasses various modifications, equivalents, and / or alternatives of the embodiments.

[0026] Figure 1 illustrates a communication environment of an electronic device according to one embodiment.

[0027] Referring to FIG. 1, according to one embodiment, an electronic device (10) may include a processor (120), a memory (130), an interface (140), a sensor (150), a display (160), a speaker (170), a camera (180), and / or a communication circuit (190). For example, the electronic device (10) may include a configuration similar to an electronic device (1401) described below with reference to FIG. 14. The electronic device (10) may be referred to as any user device. For example, the electronic device (10) may include a bar-type mobile device, a foldable electronic device (e.g., an electronic device having a foldable display), a rollable electronic device (e.g., an electronic device having a rollable display), and / or an electronic device configured to provide XR (extended reality) (e.g., a head mounted display electronic device). For example, the processor (120) may correspond to the processor (1420) of FIG. 14. The memory (130) may correspond to the memory (1430) of FIG. 14. The interface (140) may correspond to the interface (1477) of FIG. 14. The sensor (150) may correspond to the sensor module (1476) of FIG. 14. The display (160) may correspond to the display module (1460) of FIG. 14. The speaker (170) may correspond to the audio module (1470) of FIG. 14. The camera (180) may correspond to the camera module (1480) of FIG. 14. The communication circuit (190) may correspond to the communication module (1490) of FIG. 14. The configuration of the electronic device (10) illustrated in FIG. 1 is exemplary, and the configuration of the electronic device (10) is not limited thereto. For example, the electronic device (10) may further include configurations not illustrated in FIG. 1. For example, the electronic device (10) may not include at least one of the configurations illustrated in FIG. 1.

[0028] The processor (120) may be electrically, operatively, or functionally connected to memory (130), an interface (140), a sensor (150), a display (160), a speaker (170), a camera (180), and / or communication circuitry (190). In various embodiments of the present disclosure, when a component is “operatively” connected to another component, it may mean that the component is connected so as to be able to operate the other component. For example, the component may operate the other component by transmitting a control signal to the other component, either directly or via another component. In various embodiments of the present disclosure, when a component is “functionally” connected to another component, it may mean that the component is connected so as to be able to execute a function of the other component. For example, the component may execute a function of the other component by transmitting a control signal to the other component, either directly or via another component.

[0029] The processor (120) may include at least one processor. The processor (120) may include one chip or one chipset. In the present disclosure, the processor (120) may be referred to as a hardware component having an architecture by at least one processing circuit. For example, the processor (120) may be mounted on a substrate (e.g., a printed circuit board) located within the electronic device (10) and may communicate with other components of the electronic device (10) through at least one conductive path formed on the substrate.

[0030] The memory (130) can store instructions. When the instructions are individually or collectively executed by the processor (120), the instructions can cause the electronic device (10) to perform various operations. In various embodiments of the present disclosure, the operation of the electronic device (10) can be referred to as an operation performed by the processor (120) by executing instructions stored in the memory (130). The memory (130) can be referred to as a hardware component for data storage.

[0031] The interface (140) may include at least one hardware component for receiving user input. For example, the interface (140) may include a touch-sensitive circuit for receiving a touch input. For example, the interface (140) may include at least one microphone for receiving a voice input. For example, the interface (140) may include any physical interface (e.g., a knob, a jog shuttle, and / or a button) for receiving a user input.

[0032] The sensor (150) may include at least one sensor for detecting information about the surrounding environment of the electronic device (10). For example, the sensor (150) may include a light sensor, a proximity sensor, an accelerometer, a barometer, a temperature sensor, and / or a distance sensor.

[0033] The display (160) may be configured to display an image. In one example, the display (160) may include multiple displays. For example, the display (160) may include a left-eye display and a right-eye display. The display (160) may include a front display and a rear display. The display (160) may include at least one of a see-through display, a flexible display, a rollable display, a foldable display, and / or a rigid display.

[0034] The speaker (170) may be configured to output an audio signal. For example, the speaker (170) may be configured to convert an electrical signal into an acoustic signal and output the converted acoustic signal. The electronic device (10) may include a plurality of speakers. The speaker (170) may be configured to provide, for example, spatial sound.

[0035] The camera (180) may include at least one camera module configured to acquire images. For example, the processor (120) may use the camera (180) to acquire information about the surrounding environment of the electronic device (e.g., user context information).

[0036] The communication circuit (190) may include at least one circuit (e.g., a modem, a radio frequency integrated circuit, and / or a radio frequency processing circuit) configured to support communication between the electronic device (10) and a network (199) or another electronic device (e.g., an external device (20)). The communication circuit (190) may support wired communication and / or wireless communication. The communication circuit (190) may support short-range wireless communication and / or long-range wireless communication.

[0037] Figure 2 illustrates an artificial intelligence system according to one embodiment.

[0038] Referring to FIGS. 1 and 2, according to one embodiment, an electronic device (10) may be configured to implement an artificial intelligence system (200). For example, the artificial intelligence system (200) may be implemented by instructions stored in a memory (130) being executed by a processor (120). Components within the artificial intelligence system (200) may be referred to as software modules (e.g., applications, programs, or threads). The artificial intelligence system (200) illustrated in FIG. 2 is an example, and embodiments of the present disclosure are not limited thereto. For example, at least some of the components of the artificial intelligence system (200) illustrated in FIG. 2 may be implemented in an external device (20) communicatively connected to the electronic device (10). The electronic device (10) may implement the artificial intelligence system (200) by exchanging data with the external device (20) using a communication circuit (190).

[0039] For example, the artificial intelligence system (200) may include a user interface (210), an artificial intelligence framework (220), a database (230), an application / service module (240), and / or a generative AI model (250). As described above, the components of the artificial intelligence system (200) may be implemented by the electronic device (10), or at least some of the components may be implemented by the electronic device (10).

[0040] The user interface (210) may be referred to as a software interface between the artificial intelligence framework (220) and the user. For example, the user interface (210) may obtain input (e.g., user input and / or context information) by using the hardware configuration of the electronic device (10) (e.g., the interface (140), the sensor (150), and / or the camera (180)). The user interface (210) may transmit the obtained input to the artificial intelligence framework (220). The user interface (210) may be configured to output output information obtained from the artificial intelligence framework (220).

[0041] In one example, the user interface (210) may be configured to process multi-modal input. For example, the user interface (210) may receive at least one of text (e.g., natural language), images, sounds, or videos as input. In one example, the input may include input to the user interface (e.g., menu selection). For example, the user interface (210) may convert the acquired input into an input in a format supported by the artificial intelligence framework (220) and then transmit the converted input to the artificial intelligence framework (220).

[0042] In one example, the user interface (210) may transmit context information to the artificial intelligence framework (220). For example, the context information may be transmitted to the artificial intelligence framework (220) along with an input. For example, the context information may include various additional information at the time the input is received (e.g., information on the currently running application, location information, surrounding environment information acquired using a sensor (150), and / or information stored in the memory (130).

[0043] In one example, the user interface (210) may be configured to output the results of the artificial intelligence framework (220). For example, the user interface (210) may provide the results in the form of natural language, a specified content format, and / or a user-requested action. For example, the user interface (210) may be configured to output output information using the display (160) and / or the speaker (170). In one example, the user interface (210) may be configured to output the output information using visual information, auditory information, and / or tactile information.

[0044] According to one embodiment, the artificial intelligence framework (220) may be configured to receive user input and coordinate and control each component or module necessary to perform the user's intention based on the user input (e.g., a user query). For example, the artificial intelligence framework (220) may include a prompt design module (221), an application programming interface (API) / plugin management module (223), and / or a modification module (225).

[0045] For example, input (e.g., user input and / or context information) received from the user interface (210) may be transmitted to the prompt design module (221). The prompt design module (221) may be used to generate prompts suitable for inputting the input into a large language model (LLM) or a large multi-modal model (LMM). For example, the prompt design module (221) may include an artificial intelligence component that uses a machine learning algorithm or a neural network. For example, the prompt design module (221) may generate prompts from the input using the artificial intelligence component.

[0046] For example, LLM can refer to an artificial neural network-based language model that has learned a large amount of text data through pre-training. LLMs can contain significantly more parameters (e.g., over 10 billion) than conventional language models. For example, LLMs can utilize a transformer artificial neural network structure based on an attention mechanism.

[0047] The attention mechanism is a technology that helps artificial intelligence models focus (attention) on important parts of input data. The attention mechanism can be utilized to predict output data by predicting the degree to which a portion of time-series input data (e.g., input data such as voice or video, or input data of a neural network layer) contributes to the intermediate or final output of the neural network. Recurrent neural networks (RNNs), which sequentially process each element of a sequence, exhibit poor prediction performance when there is information dependency between long time-series distances. However, the attention mechanism can consider information dependency between long time-series distances by controlling the degree of weight concentration (e.g., attention) within the context of all or part of the input data.

[0048] For example, a transformer may be configured with an encoder-decoder structure. The encoder may process input data and output compressed information (e.g., a contextual representation), and the decoder may process the compressed information and output token-by-token data. Each of the encoder and decoder may include an independent attention network, or a cross-attention network connecting the encoder and decoder.

[0049] For example, LLM training may include pre-training and / or fine-tuning. Pre-training may refer to the process of teaching the LLM general linguistic knowledge using a large amount of text data. For example, pre-training may involve self-supervised learning, which predicts the next word in a text string using previous word sequences. Fine-tuning may refer to the process of training the LLM to be suitable for a specific domain (e.g., chatbot, translation, summarization, Q&A) or task. For example, the LLM may be further supervised (or adaptively trained) using a dataset tailored to the domain's purpose based on a pre-trained model. The LLM may also perform tasks based on prompts (e.g., text input containing natural language). For example, fine-tuning may be omitted from LLM training.

[0050] To enhance the performance of a user-defined task, prompts can be selected for input into the LLM. Examples of tasks and / or guidance for performing them can be additionally provided as prompts, similar to in-context learning, zero-shot learning, and / or few-shot learning.

[0051] The term "LLM" can refer to the language neural network model itself, but can also refer to the model of an LLM-based application (e.g., chatbot, translation, summarization, text classification, sentence generation). For example, an LLM-based chatbot like ChatGPT or an LLM-based translator can also be referred to as an "LLM." Publicly available LLMs include BERT (bidirectional encoder representations from transformer) and GPT (generative pre-trained transformer).

[0052] "LLM" may also include an inference engine utilizing the LLM neural network model. For example, "entering an input prompt into the LLM" may mean "entering the input prompt into an inference engine based on the LLM." For example, "the output of the LLM for the input prompt" may refer to the output information of the last neural network layer of the LLM obtained when the input prompt is entered into the LLM-based inference engine.

[0053] The prompt design module (221) may be configured to generate prompts based on input and / or a database (230). For example, the database (230) may store user preference data, a prompt library, and / or prompt examples. For example, the prompt design module (221) may generate prompts using information stored in the database (230). As described below, the prompt design module (221) may use information obtained using the API / plugin management module (223) to generate prompts. The generated prompts may be passed to a generative AI model (250) (e.g., a large language model (LLM) or a large multi-modal model (LMM)).

[0054] The API / plug-in management module (223) may be configured to exchange data with an external entity of the artificial intelligence framework (220). For example, the API / plug-in management module (223) may create a channel for communication with an external entity (e.g., an operating system and / or an application) of the artificial intelligence framework (220) via an API. The API / plug-in management module (223) may access various data sources to acquire data via the created channel. The data acquired by the API / plug-in management module (223) may be used to generate a prompt by the prompt design module (221). The data acquired by the API / plug-in management module (223) may be used as an input for the generative AI model (250). In one example, when data (e.g., user input) is required in the process of generating a result of the artificial intelligence framework (220), the API / plug-in management module (223) may acquire the data using an application and / or service of the application / service module (240).

[0055] In one example, the application / service module (240) may include at least one application and / or at least one service. For example, the at least one service may include an operating system service. The applications and / or services of the application / service module (240) may communicate with the API / plugin management module (223) via an API or a plug-in. The API / plugin management module (223) may transmit data instructing a specified operation to the application or service module of the application / service module (240) via the API or plug-in.

[0056] In one embodiment, the modification module (225) can fine-tune the output from the generative AI model (250). For example, the modification module (225) can filter content generated by the generative AI model (250). The modification module (225) can be configured to remove or filter biased content, harmful content, content with low relevance to the prompt, and / or aversive content from the output results. In one example, the modification module (225) can identify a correlation (e.g., a correlation) between the output results of the generative AI model (250) and a user's intention (e.g., an intention identified from user input and / or context information). If the correlation between the output results and the user's intention is low, the modification module (225) can perform additional operations. For example, the modification module (225) can obtain additional input from the user to obtain a result with higher relevance. To prevent repetition of additional procedures, the modification module (225) can generate a hint and provide the generated hint to the user.

[0057] According to one embodiment, a generative AI model (250) may be referred to as an artificial intelligence neural network that generates new types of data (e.g., data that includes information not included in the input information) from user input information. The generative AI model (250) may include a model that generates images and / or a model that generates language. The model that generates images may include, for example, a generative adversarial network (GAN), a variational auto encoder (VAE), and / or a diffusion-based generative model that uses a VAE and a transformer architecture. The model that generates language may include a model trained to output statistically most appropriate output values ​​based on input values. For example, the model that generates language may include a model such as CHAT-GPT 3 or CHAT-GPT 4. The generative AI model (250) may include a large language model (LLM) or a large multi-modal model (LMM).

[0058] Figure 3 illustrates modules of an electronic device according to one embodiment.

[0059] Referring to FIGS. 1 and 3, according to one embodiment, the electronic device (10) may be configured to implement a UI generation system (300). For example, the UI generation system (300) may be implemented by instructions stored in the memory (130) being executed by the processor (120). Components within the UI generation system (300) may be referred to as software modules (e.g., applications, programs, or threads). The components of the UI generation system (300) illustrated in FIG. 3 are merely an example, and embodiments of the present disclosure are not limited thereto. For example, at least some of the components of the UI generation system (300) illustrated in FIG. 3 may be implemented in an external device (20) communicatively connected to the electronic device (10). The electronic device (10) may implement the UI generation system (300) by exchanging data with the external device (20) using a communication circuit (190).

[0060] According to one embodiment, the UI generation system (300) may include a data management module (310), a trigger management module (320), a UI generation module (330), a UI output module (340), a database (390), an application (3010), and / or an operating system (OS) (3020). The application (3010) may include, for example, at least one application installed in the electronic device (10). The OS (3020) may include the OS of the electronic device (10). The application (3010) and the OS (3020) may be separate components from the UI generation system (300).

[0061] The data management module (310) can collect and store data. The data management module (310) can obtain data from the application (3010) and / or the OS (3020) and store the obtained data in the database (390). For example, the data management module (310) can obtain data from the application (3010) and / or the OS (3020) using the API / plug-in management module (223) described above with reference to FIG. 2 . In one example, the data management module (310) can obtain message information from the message application of the electronic device (10) and store the obtained information in the database (390). In one example, the data management module (310) can obtain schedule information from the schedule application of the electronic device (10) and store the obtained information in the database (390). In one example, the data management module (310) may acquire configuration information (e.g., installed application information) of the OS (3020) and store the acquired information in a database (390). The data management module (310) may structure the acquired information and store it in the database (390). The operation of the data management module (310) may be described later with reference to FIG. 4.

[0062] The trigger management module (320) may be configured to determine whether to generate a recommendation UI. For example, the trigger management module (320) may determine whether to generate a recommendation UI using information from the database (390) and / or information obtained from the OS (3020). The trigger management module (320) may identify a recommendation action based on the obtained information and determine whether to generate a recommendation UI that notifies the identified recommendation action. If generation of a recommendation UI is determined, the trigger management module (320) may cause the UI generation module (330) to generate the recommendation UI. The operation of the trigger management module (320) may be described below with reference to FIG. 6 .

[0063] The UI generation module (330) may be configured to generate a UI. For example, the UI generation module (330) may generate a UI by inputting a generated prompt into a generative AI model (e.g., the generative AI model (250) of FIG. 2 ). In one example, the UI generation module (330) may generate a prompt based on information received from the trigger management module (320). The operation of the UI generation module (330) may be described below with reference to FIG. 7 .

[0064] The UI output module (340) may be configured to output the UI generated by the UI generation module (330). For example, the UI output module (340) may output the generated UI through an external electronic device connected via the display (160) or communication circuit (190) of the electronic device (10). For example, the UI output module (340) may correspond to the user interface (210) and / or the modification module (225) of FIG. 2. A description of the UI output module (340) may be provided below with reference to FIG. 7.

[0065] According to one embodiment, the electronic device (10) may be configured to provide a UI including information on recommended actions to the user using the UI generation system (300). The electronic device (10) may collect information from an application (3010) and / or an OS (3020) using the data management module (310). The electronic device (10) may identify an intent and / or a goal from the collected information using the trigger management module (320), and may identify a subsequent action from the intent and / or goal. For example, the electronic device (10) may identify an action that the user is expected to perform in the future and / or additional information (e.g., additional information related to the subsequent action) by analyzing information (e.g., text, images, and / or videos) displayed on the display (160) of the electronic device (10). For example, the electronic device (10) may analyze a conversation between the user and the other party through a messenger application in real time. The electronic device (10) can determine the need to provide a recommendation using the trigger management module (320).

[0066] If it is determined that a recommendation is required, the electronic device (10) can generate a recommendation UI using the UI generation module (330). The electronic device (10) can generate a prompt associated with a subsequent action and / or a recommendation provision using the UI generation module (330). The UI generation module (330) can generate a recommendation UI by inputting the prompt into a generative AI model (e.g., the generative AI model (250) of FIG. 2). The electronic device (10) can generate a recommendation UI including a subsequent action and information associated with the subsequent action.

[0067] The electronic device (10) can provide (e.g., display) the generated recommendation UI. For example, the electronic device (10) can display the recommendation UI on at least a portion of the display (160) using the UI output module (340). The recommendation UI can include, for example, at least one of a graphic object, a floating window, a pop-up window, an in-application message, a widget, or an icon. In one example, the electronic device (10) can transmit information about the recommendation UI to an external device (e.g., a smartwatch, a wearable device, a tablet, a desktop, a laptop, a TV, and / or an XR (extended reality) device) using the communication circuit (190), thereby providing the recommendation UI through the external device.

[0068] FIG. 4 illustrates a data management module of an electronic device according to one embodiment.

[0069] Referring to FIGS. 1 and 4, according to one embodiment, the data management module (310) may be configured to collect information from a service of the electronic device (10) (e.g., an application (3010) and / or an OS (3020)). In one example, the data management module (310) may collect information from a running application (3010) and / or an OS (3020) of the electronic device (10). The data management module (310) may collect information based on a specified event (e.g., input reception) and / or a specified cycle. The data management module (310) may store information stored in the memory (130) of the electronic device (10), information acquired by an application (e.g., received messages, notifications, and / or user input), and / or information acquired using a sensor (150) in a database (390). The data management module (310) may include a data receiving module (410) and a data collection module (420).

[0070] The data receiving module (410) can receive data from the application (3020) and / or the OS (3020). The data receiving module (410) can receive data through a designated interface (e.g., API). Each of the first application (3011), the second application (3012), and / or the third application (3013) can transmit designated information to the data receiving module (410) using a designated software development kit (SDK). For example, the data receiving module (410) can receive data through an interface defined by the SDK. The data can be transmitted to the data receiving module (410) after being processed into a data format designated by the SDK. In one example, the data receiving module (410) can receive data through an interface (e.g., API) provided by the OS (3020). The data receiving module (410) can store the received data in a database (390).

[0071] The data collection module (420) can collect information through an interface (e.g., API) provided by the OS (3020). The data collection module (420) can be configured to collect any type of data (e.g., text, images, and / or videos). For example, the data collection module (420) can obtain text information and / or image information within a displayed screen (e.g., a page) through a contents capture service. In one example, the data collection module (420) can obtain text information and / or image information from the displayed screen based on a text recognition and / or image recognition algorithm. For example, the data collection module (420) can obtain information about a notification of the electronic device (10) (e.g., an application associated with the notification and / or the content of the notification) using a notification service. For example, the data collection module (420) can obtain information (e.g., type of media, title of media, and / or media playback application) of media (e.g., image and / or video) currently being played on the electronic device (10) through a media service. For example, the data collection module (420) can obtain information about a running application (e.g., application name and / or execution environment of the application (e.g., foreground or background)) by using an application usage receive service. The data collection module (420) can store the collected data in a database (390). For example, the data collection module (420) can process the collected data into a specified data format and then store the data in the specified data format in the database (390).

[0072] The data management module (310) can store data in a specified data format in the database (390). For example, the specified data format may include structured text (e.g., json and / or jsonl). For example, the specified data format may include a source, a type, and information about the data (e.g., text information extracted from text, images, and / or videos). The source may include information about the entity that provided the data (e.g., the name of the application that provided the data). The type may include, for example, a composite type and a primitive type. Composite data may be referred to as data in which multiple types of data are mixed. Primitive data may be referred to as a single type of data. In one example, the type information of primitive data may include type information of the data (e.g., event, message, and / or media). The data may include detailed information according to the type information of the data. For example, the data may be configured in a preset format (e.g., field, field order, and / or details) according to the type information of the data. For example, the detailed information may include at least a portion of extracted text information, at least a portion of text information extracted based on image recognition (e.g., keywords of an image extracted based on image recognition), and / or at least a portion of text information extracted based on recognition of images in a video.

[0073] In the above example, a database (390) containing structured text is described, but embodiments of the present disclosure are not limited thereto. For example, the data collection module (420) may store collected text, images, and / or videos in the data collection module (420). The data collection module (420) may store, for example, text information associated with the displayed image (e.g., structured text) together with an image (e.g., a displayed image) mapped to the text information in the database (390). The data collection module (420) may store, in the database (390), an image (e.g., an image of the video being played) and / or a video (e.g., a video being played) mapped to the text information together with text information associated with the video being played (e.g., structured text).

[0074] For example, the data management module (310) can obtain schedule information from a calendar application. The data management module (310) can obtain schedule information when a schedule is entered or when a specified period arrives. The data management module (310) can store the schedule information in a database (390) as data in a specified format. For example, the source of the schedule information can be set to a calendar application (e.g., application package name). The type of the schedule information can be set to an event. The data of the schedule information can include, for example, at least one of a start time, an end time, a title, a location, or schedule attendee information (e.g., attendee name, email, and / or phone number).

[0075] For example, the data management module (310) can obtain message information from a messaging application. The data management module (310) can obtain message information when a message is received, when a message is sent, or when a specified period has elapsed. The data management module (310) can store the message information in a database (390) as data in a specified format. For example, the source of the message information can be set to a messaging application (e.g., an application package name). The type of the message information can be set to a message. The data of the message information can include, for example, at least one of a conversation type, other party information (e.g., name, email, and / or phone number), or message content. The conversation type can include a one-on-one conversation or a group conversation. For example, the data management module (310) can store some of the content included in the message as message content. The data management module (310) may identify at least one keyword (e.g., intent and / or entity) from the content of a message using, for example, an artificial intelligence model (e.g., LLM) and store the identified keyword as message content. In one example, the message may include an image. The data management module (310) may identify at least one keyword (e.g., a keyword extracted based on the recognition of an object within the image) from the image and store the identified keyword as message content.

[0076] The database (390) can store data in a specified format. The database (390) can include, for example, a knowledge-based graph database. For example, the database (390) can store information about relationships between data. The data within the database (390) can be used to determine triggers and generate prompts, as described below.

[0077] FIG. 5 illustrates a message screen of an electronic device according to one embodiment.

[0078] Referring to FIGS. 1, 3, and 5, the message application of the electronic device (10) may display a message screen (501) on the display (160). For example, a first received message (511) may include information about a recommendation of a movie (e.g., AAA). A second received message (512) may include information about the recommended movie (e.g., an image of the movie). A first sent message (521) may include information indicating whether a user (e.g., a user of the electronic device (10)) has watched the recommended movie (e.g., not watched). A second sent message (522) may include an inquiry about information about the recommended movie. A third received message (513) may include a rating for the recommended movie. A third sent message (523) may include a user's reaction to the recommended movie.

[0079] According to one embodiment, the electronic device (10) can identify at least one keyword from the conversation of FIG. 5. For example, the electronic device (10) can identify at least one keyword using the data management module (310). The electronic device (10) can identify at least one keyword from the conversation of FIG. 5 using an artificial intelligence model (e.g., LLM). For example, the electronic device (10) can identify “AAA,” a keyword corresponding to information about a recommended movie, from the first received message (511) and / or the second received message (512). The electronic device (10) can identify keywords such as “not yet,” “absolutely,” and “got to check it out” from the first sent message (521), the third received message (513), and the third sent message (523). In one example, the electronic device (10) can identify an intent to watch the movie “AAA” from the identified keywords.

[0080] In one example, the electronic device (10) may store identified keywords and / or intents in a database (390). The electronic device (10) may store each message as separate structured data in the database (390). The electronic device (10) may store messages transmitted and received within a specified time period as a single structured data in the database (390).

[0081] In the example of FIG. 5, the source of the structured data may be set to a message application, and the type may be set to message. The data fields of the structured data may include at least one identified keyword and / or intent. In one example, the electronic device (10) may identify additional information from the at least one keyword and / or intent and store the identified additional information in the structured data. For example, from the identified intent of “watching AAA,” the electronic device (10) may identify information about an application (e.g., a content provider application) capable of watching AAA and store the identified information in the structured data. For example, if the recommended “AAA” is a TV series show, the electronic device (10) may identify information about the latest episode of the show and store the latest episode information in the structured data.

[0082] FIG. 6 illustrates a trigger management module of an electronic device according to one embodiment.

[0083] Referring to FIGS. 1 and 6, according to one embodiment, the trigger management module (320) may determine the triggering of the generation of a recommendation UI. For example, the trigger management module (320) may determine the triggering of the generation of a recommendation UI using information stored in a database (390) or information obtained from an OS (3020). For example, the trigger management module (320) may include a trigger data handler (610), a data analyzer (620), a trigger resolver (630), and / or a trigger database (640).

[0084] The trigger data handler (610) can obtain data from the database (390) and / or the OS (3020) and transmit the obtained data to the data analyzer (620) or the trigger resolver (630). The trigger data handler (610) can perform data transmission between the data analyzer (620) and the trigger resolver (630). The trigger data handler (610) can transmit information for UI generation to the UI generation module (330). The trigger data handler (610) can be referred to as an interface between the trigger management module (320) and other components.

[0085] The trigger resolver (630) may determine the triggering of the generation of a recommendation UI using information received from the trigger data handler (610) (e.g., information obtained from the database (390) and / or the OS (3020). If the triggering of the generation of a recommendation UI associated with specific data is determined, the trigger resolver (630) may transmit the relevant information to the data analyzer (620) so that a recommendation action associated with the specific data may be generated. If the triggering is determined not to occur, the electronic device (10) may stop performing the operations described below and continue to collect information as described above.

[0086] For example, the trigger resolver (630) can obtain schedule information stored in the database (390). The trigger resolver (630) can determine (e.g., trigger) the generation of a recommendation UI if the start time of the stored schedule is within a specified time from the current time. In one example, the trigger resolver (630) can determine the generation of a recommendation UI based on a policy stored in the trigger database (640). The trigger database (640) can store, for example, information on conditions (e.g., policies) for the generation of a recommendation UI. The trigger database (640) can store information on conditions for the generation of a plurality of recommendation UIs mapped to each data source or type.

[0087] For example, the trigger resolver (630) may obtain message information stored in the database (390). When a keyword indicating the user's intention and / or willingness is identified from the message information, the trigger resolver (630) may determine to generate a recommendation UI associated with the corresponding message information. For example, keywords indicating the user's intention may include keywords indicating the user's intention to perform an action. Keywords indicating the user's willingness may include keywords indicating agreement with the other party's recommended action (e.g., keywords indicating strong affirmation) and / or keywords indicating the user's willingness to perform the recommended action. In the example of FIG. 5, the trigger resolver (630) may determine to generate a recommendation UI based on keywords such as "absolutely" and / or "got to check it out."

[0088] Although the example of FIG. 5 illustrates an example in which the trigger resolver (630) determines the generation of a recommendation UI based on a conversation between a user and a counterpart, embodiments of the present disclosure are not limited thereto. For example, the trigger resolver (630) may determine the generation of a recommendation UI based on information stored internally in the electronic device (10) and / or external search information. In the example of FIG. 5, the trigger resolver (630) may identify a recommendation target called the movie 'AAA'. In one example, the trigger resolver (630) may retrieve the movie AAA from information stored in the electronic device (10). For example, the trigger resolver may obtain information about AAA (e.g., conversations and / or schedules related to a keyword (e.g., AAA)) from schedule information and / or other conversations (e.g., conversations with other parties than the example of FIG. 5) stored in the electronic device (10). The trigger resolver (630) may determine generation of a recommendation UI using the information stored in the electronic device (10). In one example, the trigger resolver (630) may search the web for information about 'AAA' and determine generation of a recommendation UI based on the retrieved information. For example, the trigger resolver (630) may obtain the popularity of AAA (e.g., rating, reservation rate, search frequency, and / or search rank) from a web-based database, If the popularity is above a specified value, you can decide to create a recommendation UI.

[0089] The data analyzer (620) can analyze information received from the trigger data handler (610) (e.g., information obtained from the database (390) and / or the OS (3020)). For example, the data analyzer (620) can identify a user's intention from the information received from the trigger data handler (610). The data analyzer (620) can obtain at least one keyword from data in a specified format and identify an intention from the obtained keyword. For example, the data analyzer (620) can identify an intention from a keyword using an artificial intelligence model (e.g., LLM).

[0090] The data analyzer (620) can identify intentions and recommended actions based on information (e.g., information obtained from the database (390) and / or the OS (3020). For example, the data analyzer (620) can obtain recommended actions by inputting identified intentions and identified keywords into an artificial intelligence model. For example, the data analyzer (620) can obtain intentions and recommended actions by inputting identified keywords into an artificial intelligence model. The recommended actions can include a series of actions for realizing a goal corresponding to the intention. The recommended actions can include generating a recommended UI for realizing the goal corresponding to the intention.

[0091] The data analyzer (620) may obtain additional information based on information (e.g., information obtained from the database (390) and / or the OS (3020). For example, the data analyzer (620) may obtain information related to keywords, intentions, purposes, and / or recommended actions through an external server (e.g., a search server). The data analyzer (620) may obtain recommendation information related to keywords, intentions, purposes, and / or recommended actions, and include the obtained recommendation information in the information regarding the recommended actions.

[0092] The trigger data handler (610) may transmit information about the recommended action generated by the data analyzer (620) to the UI generation module (330). For example, the information about the recommended action generated by the data analyzer (620) may include structured data (e.g., json or jsonl). The information about the recommended action may include purpose information, type information, and / or data information. The purpose information may include information about the purpose identified by the trigger management module (320). For example, if the recommended action is to generate a recommended UI associated with the playback of a specific media, the purpose information may be set to generate the UI. The type information may indicate the purpose of the recommended action (e.g., the purpose of the final action (generating the recommended UI)). If the recommended action is to generate a recommended UI associated with the playback of a specific media, the type information may be set to media playback. The data information may include information related to data associated with the recommended action. If the recommended action is to generate a recommended UI associated with the playback of a specific media, the data information may include information related to the specific media. For example, the data information may include information about an application for playing a particular media, a name of the particular media, and / or additional information (e.g., episode information).

[0093] In the example of FIG. 5, the data analyzer (620) can identify the intent of watching AAA from the conversation. The data analyzer (620) can obtain information on an application capable of playing AAA (e.g., a content provider application) using the AAA information. The data analyzer (620) can identify the purpose of playing AAA through the application from the intent of watching AAA. The data analyzer (620) can identify a recommended action to achieve the identified purpose. The recommended action can include generating a recommended UI that causes AAA to be played through the application. The data analyzer (620) can set the purpose information of the recommended action information to create a UI, the type information of the recommended action information to play media, and the data information to information related to the movie AAA (e.g., the movie name and / or information on an application capable of playing the movie).

[0094] FIG. 7 illustrates a UI generation module of an electronic device according to one embodiment.

[0095] Referring to FIGS. 1 and 7, according to one embodiment, the electronic device (10) may generate at least one prompt based on recommended action information. The electronic device (10) may generate a UI based on the at least one prompt. For example, the electronic device (10) may generate the prompt and UI using a UI generation module (330). For example, the UI generation module (330) may include a UI handler (710), a prompt generation module (720), a generative AI model (730), a UI database (740), and / or a UI provision module (750).

[0096] The UI handler (710) can control data exchange within and outside the UI generation module (330) and the operation of the UI generation module (330). For example, the UI handler (710) can receive recommended action information from the trigger management module (320) of FIG. 3 and transmit the received recommended action information to the prompt generation module (720). For example, the UI handler (710) can obtain information from the application (3010), the OS (3020), and / or an external database (790). The external database (790) can be referenced as, for example, a database of any electronic device located outside the electronic device (10).

[0097] The prompt generation module (720) (e.g., the prompt design module (221) of FIG. 2 ) may generate at least one prompt based on recommended action information. For example, the prompt generation module (720) may generate at least one prompt by processing the recommended action information using an artificial intelligence model (e.g., LLM) or a generative AI model. In one example, the prompt may have a format that can be processed by the generative AI model.

[0098] The generative AI model (730) may be configured to perform at least one action based on a prompt. The generative AI model (730) may include, for example, multiple generative AI models configured to perform a specified action. The generative AI model (730) may include a generative adversarial network (GAN), a variable-average electric field (VAE), or a style GAN. The generative AI model (730) may be configured to perform data acquisition, UI generation, and / or response generation.

[0099] For example, the prompt generation module (720) may generate a prompt to collect additional information related to the recommended action information (e.g., information necessary for performing the recommended action). The prompt generation module (720) may generate a prompt to obtain additional information related to the recommended action information from an external database (790) using information included in the recommended action information. The generative AI model (730) may collect additional information from, for example, the database (390), the application (3010), the OS (3020), and / or the external database (790) of FIG. 3.

[0100] For example, the prompt generation module (720) can generate a prompt for collecting information of a user and / or an electronic device (10). The prompt generation module (720) can generate a prompt for obtaining hardware information of the electronic device (10) necessary for performing a recommended action. The generative AI model (730) can obtain required user information and / or information of the electronic device (10) (e.g., resolution information of the display (160) and / or network connection information of the communication circuit (190)) from the database (390), application (3010), OS (3020), and / or external database (790) of FIG. 3.

[0101] For example, the prompt generation module (720) can generate a prompt for generating a structure of a recommended UI (e.g., size of the UI, location of the UI, components within the UI, locations of components, component-related actions, and / or UI-related color information). The prompt generation module (720) can generate a prompt based on previously acquired additional information, user information, and / or information of the electronic device (10). The generative AI model (730) can generate a structure of a recommended UI based on the prompt. In one example, the generative AI model (730) can be configured to generate a recommended UI corresponding to the prompt by using a plurality of reference UIs stored in the UI database (740).

[0102] For example, the prompt generation module (720) may generate a prompt to obtain a UI resource. The generative AI model (730) may be configured to obtain the UI resource indicated by the prompt from the UI database (740), the application (3010), the OS (3020), and / or the external database (790).

[0103] According to one embodiment, the UI providing module (750) can generate a recommendation UI based on the acquired UI resources and the structure of the UI. For example, the UI providing module (750) can generate a recommendation UI by mapping the acquired UI resources to the structure of the UI. The recommendation UI can include information corresponding to a recommended action (e.g., information for executing the recommended action). For example, the UI providing module (750) can generate a recommendation UI according to a widget processing flow. The UI providing module (750) can generate a recommendation UI in the form of a widget (e.g., an in-app message) or a pop-up widget (e.g., a floating view) within an application.

[0104] The UI output module (340) may be configured to output the recommended UI generated by the UI generation module (330). For example, the UI output module (340) may output the recommended UI through the display (160) of the electronic device (10). For example, the UI output module (340) may transmit information about the recommended UI using the communication circuit (190) to cause the external device (20) to output the recommended UI. For example, the UI output module (340) may correspond to the modification module (225) of FIG. 2.

[0105] Figure 8 is a flowchart of a method for generating a prompt in an electronic device according to one embodiment.

[0106] Referring to FIG. 8, the operations described below with respect to FIG. 8 may be referred to as operations of the electronic device (10) of FIG. 1. The order of the operations described below with respect to FIG. 8 is merely an example, and embodiments of the present disclosure are not limited thereto. For example, at least some of the operations may be executed differently from the order of FIG. 8, or may be executed substantially simultaneously with other operations of FIG. 8. At least some of the operations described below with respect to FIG. 8 may be omitted. Unless otherwise stated, the operations described above with respect to FIG. 7 may be referred to in the description of FIG. 8.

[0107] In the example of Fig. 8, the recommended action may be assumed to be the creation of a recommendation UI for playing a movie AAA using the M application. For example, the electronic device (10) may identify the recommended action according to the example of Fig. 5.

[0108] In operation 805, the electronic device (10) can determine whether the target application (e.g., M application) is already installed. For example, the electronic device (10) can obtain information on the installation application from the database (390) or the OS (3020) of FIG. 3, and determine whether to install the M application based on the obtained information. For example, the electronic device (10) can generate a prompt (e.g., “Check whether the M application is installed”) to confirm whether the M application is installed using the prompt generation module (720). The electronic device (10) can obtain a result value by inputting the prompt to the generative AI model (730), and determine whether to install the M application based on the result value. Information on the installation of the target application can be included in additional information, user information, or information of the electronic device (10) obtained by the generative AI model (730) prior to operation 805.

[0109] If the target application is not installed (e.g., operation 805-NO), in operation 810, the electronic device (10) may generate a prompt for generating a UI associated with the installation of the target application. For example, if the M application is not installed, the prompt generation module (720) may generate a prompt that instructs the generation of a recommendation UI recommending the installation of the M application (e.g., generate an M application installation suggestion UI). In one example, the prompt generation module (720) may generate multiple prompts. A first prompt may instruct the generation of an installation suggestion UI for the M application, a second prompt may instruct the background color of the UI, a third prompt may instruct the position and size of a graphical object corresponding to the installation shortcut, and a fourth prompt may instruct a shortcut (e.g., an installation link for the application) and / or a search (e.g., a search within the store for the application) operation for the graphical object. The multiple prompts may be input to the generative AI model (730) at the same time or may be input to the generative AI model (730) sequentially. When multiple prompts are input simultaneously, the generative AI model (730) can generate a recommendation UI based on the multiple prompts. In one example, after a result based on the first prompt is generated, subsequent prompts can be sequentially input to the generative AI model (730). The subsequent prompts can be processed by the generative AI model (760) as additional commands to the results of the preceding prompt.

[0110] The UI output module (340) can display a recommendation UI recommending the installation of the M application. For example, the prompt generation module (720) can generate a prompt to obtain graphic resources within the UI by referencing the UI database (740). The generative AI model (730) can obtain graphic resources based on the prompt and generate a UI based on the obtained graphic resources and the UI structure. The UI output module (340) can output the generated UI. If the M application is installed based on the input for the recommendation UI recommending the installation of the M application, the electronic device (10) can perform the subsequent operation 815.

[0111] If the target application is installed (e.g., operation 805-YES), in operation 815, the electronic device (10) may determine whether an access restriction exists. For example, the electronic device (10) may determine whether login and / or authentication for access is required to perform the recommended operation. For example, the electronic device (10) may obtain the login information of the target application from the database (390), the application (3010), or the OS (3020) of FIG. 3, and determine whether an access restriction exists based on the obtained information. For example, the electronic device (10) may use the prompt generation module (720) to generate a prompt to confirm whether the M application is logged in (e.g., “Please confirm whether the M application is logged in”). The electronic device (10) may input the prompt into the generative AI model (730) to obtain a result value, and determine whether the M application is logged in based on the result value. Information about the login of the target application may be included in additional information, user information, or information of the electronic device (10) obtained by the generative AI model (730) prior to operation 815.

[0112] If there is an access restriction (e.g., operation 815-YES), in operation 820, the electronic device (10) may generate a prompt for generating a UI related to receiving access restriction-related information. For example, if a login to the M application is required, the prompt generation module (720) may generate a prompt that instructs generation of a recommendation UI recommending login to the M application (e.g., generate an M application login suggestion UI). In one example, the prompt generation module (720) may generate multiple prompts. A first prompt may instruct generation of a login suggestion UI for the M application, a second prompt may instruct the background color of the UI, a third prompt may instruct the position and size of a graphic object corresponding to an application execution link, and a fourth prompt may instruct a shortcut (e.g., an application execution link) operation (e.g., setting an intent or deep link) for a graphic object. As described above, multiple prompts may be input to the generative AI model (730) at the same time or sequentially.

[0113] The UI output module (340) can display a recommended UI recommending login to the M application. For example, the prompt generation module (720) can generate a prompt to obtain graphic resources within the UI by referencing the UI database (740). The generative AI model (730) can obtain graphic resources based on the prompt and generate a UI based on the obtained graphic resources and the UI structure. The UI output module (340) can output the generated UI. If the user logs into the M application based on an input for the recommended UI recommending login to the M application, the electronic device (10) can perform the subsequent operation 825.

[0114] With respect to operation 820, it has been described that a UI is provided to input information for connection when access restrictions exist; however, embodiments of the present disclosure are not limited thereto. For example, the UI output module (340) may output a recommended UI generated based on currently accessible information (e.g., information that can be accessed without logging in). The UI output module (340) may output a UI associated with content execution based on intent (e.g., a UI generated according to operation 830) based on currently accessible information.

[0115] If there is no access restriction (e.g., operation 815-NO), in operation 825, the electronic device (10) may determine whether additional information exists. For example, the electronic device (10) may obtain the additional information from the database (390), the application (3010), or the OS (3020) of FIG. 3. For example, the electronic device (10) may generate a prompt for obtaining the additional information using the prompt generation module (720). The electronic device (10) may obtain a result by inputting the prompt into the generative AI model (730), and may obtain the additional information based on the result.

[0116] If no additional information exists (e.g., operation 825-NO), in operation 835, the electronic device (10) may generate a prompt for generating a UI associated with content execution based on the intent. For example, the prompt generation module (720) may generate a prompt for generating a UI structure and a prompt for acquiring UI resources. The generative AI model (730) may generate a recommendation UI based on the generated prompts. The generated recommendation UI may include information for playing the content (e.g., application execution, media playback, and / or a link to a specified view). For example, the information for playing the content may include a deep link action URI (uniform resource indicator).

[0117] For example, prompts for generating a structure may include prompts that instruct the location of an application icon, the size of an application icon, the background color of a UI, the location of a thumbnail, the size of a thumbnail, the size of a graphical object, the location of a graphical object, and / or actions associated with a graphical object. For example, prompts for acquiring UI resources may include prompts to acquire graphical resources within the UI by referencing a UI database (740) or an external database (790).

[0118] In one embodiment, the generative AI model (730) can generate a recommendation UI based on a prompt. The prompt may omit some information about the recommendation UI. For example, the prompt may not include information about the size or location of the recommendation UI. The generative AI model (730) can supplement the missing information using information stored in the UI database (740).

[0119] The generated recommendation UI can be output by the UI output module (340). Based on an input for the recommendation UI (e.g., an input instructing to terminate the display of the recommendation UI or an input instructing to perform a recommendation action instructed by the recommendation UI), the UI output module (340) can terminate the output of the recommendation UI. Hereinafter, various output examples of recommendation UIs can be described with reference to FIGS. 9A to 11.

[0120] FIG. 9a illustrates a recommendation UI of an electronic device according to one embodiment.

[0121] Referring to FIGS. 1 to 8 and 9A, according to one embodiment, the electronic device (10) may display a recommendation UI (910). The first screen (901) of FIG. 9A may be a subsequent screen to the message screen (501) of FIG. 5. For example, the electronic device (10) may display the recommendation UI (910) on the execution screen of a running application or as an in-app message of the running application.

[0122] For example, the electronic device (10) can generate a recommendation UI (910). In one example, the recommendation UI (910) can include a UI generated based on a prompt generated according to operation 830 of FIG. 8. The recommendation UI (910) can include, for example, additional information (920), a thumbnail (930) of the movie AAA, a link image (940) of the M application, and / or a title (950). For example, the electronic device (10) can obtain the additional information (920), the thumbnail (930), the link image (940), and / or the title (950) from the database of the M application. For example, the electronic device (10) can generate a prompt for obtaining a UI resource using the prompt generation module (720) of FIG. 7, and input the generated prompt into the generative AI model (730) to obtain the UI resource. For example, in response to an input for a thumbnail (930), the electronic device (10) may be configured to play a movie corresponding to the thumbnail (930). For example, in response to an input for a link image (940), the electronic device (10) may be configured to execute an application corresponding to the link image (940). For example, the actions of the electronic device (10) related to an input for a thumbnail (930) and / or a link image (940) may be configured based on a prompt related to the generation of the structure described above.

[0123] In one example, the prompt generation module (720) may generate a prompt indicating the type of output to be made of the recommendation UI (910). For example, the prompt may indicate that the recommendation UI (910) is to be output (e.g., immediately output) on the execution screen of the currently running application. For example, the recommendation UI (910) may be output in the form of a widget, an in-app message, or a floating window. For example, the generative AI model (730) may generate the recommendation UI (910) according to the output type indicated by the prompt.

[0124] In one example, the generative AI model (730) may determine the output type of the recommendation UI (910) based on contextual information. The generative AI model (730) may determine the output type based on information of the user and / or information of the electronic device (10), even if the output type is not indicated by a prompt. For example, if the moving speed of the electronic device (10) is below a threshold or the electronic device (10) is located at a designated location (e.g., home, work, or school), the generative AI model (730) may cause the recommendation UI (910) to be output (e.g., immediately output) on the execution screen of the currently running application.

[0125] FIG. 9b illustrates a recommendation UI of an electronic device according to one embodiment.

[0126] Referring to FIGS. 1 to 8 and 9B, according to one embodiment, the electronic device (10) may display a second screen (902). The second screen (902) of FIG. 9B may be a subsequent screen to the message screen (501) of FIG. 5. For example, the electronic device (10) may display the recommendation UI on the execution screen of the running application, or as an in-app message of the running application.

[0127] In the example of FIG. 9B, the recommendation UI may be composed of a link image (940) of an M application that allows viewing of the movie AAA. For example, the electronic device (10) may recommend an action to view the movie AAA recommended by the second received message (512) through the M application by displaying the link image (940) adjacent to the second received message (512). As described above with respect to FIG. 9A, the electronic device (10) may display the link image (940) on the execution screen of the currently executing application. For example, the electronic device (10) may determine the display timing of the link image (940) based on the user's context (e.g., intent identified by the context). The electronic device (10) may display the link image (940) when a positive intent of the user, such as the third sent message (523), is identified. For example, the electronic device (10) may display a link image (940) when a second received message (512) is received.

[0128] With reference to FIGS. 9A and 9B , examples of providing a recommendation UI on a currently displayed execution screen by an electronic device (10) have been described. However, embodiments of the present disclosure are not limited thereto. For example, as described below with reference to FIGS. 9C and 9D , the electronic device (10) may provide a recommendation UI after the display of the currently displayed execution screen ends.

[0129] FIG. 9c illustrates a recommendation UI of an electronic device according to one embodiment.

[0130] Referring to FIGS. 1 to 8 and 9C, according to one embodiment, the electronic device (10) may display a recommendation UI (910). The third screen (903) of FIG. 9C may correspond to a screen (e.g., a home screen of the electronic device (10)) displayed after the display of the message screen (501) of FIG. 5 is terminated. For example, the electronic device (10) may be set to display the recommendation UI (910) after the display of the execution screen of the application (e.g., a message application) that triggered the recommendation action is terminated.

[0131] For example, the electronic device (10) may generate a recommendation UI (910) based on a prompt generated according to operation 830 of FIG. 8. The generation operation of the recommendation UI (910) may refer to the contents described above with respect to FIG. 9A, unless otherwise described.

[0132] In one example, the prompt generation module (720) may generate a prompt indicating the type of output to be made of the recommendation UI (910). For example, the prompt may indicate that the recommendation UI (910) is to be output (e.g., delayed output) when the display of the execution screen of the currently running application is terminated. For example, the recommendation UI (910) may be output in the form of a widget, a pop-up, or a floating window. For example, the generative AI model (730) may generate the recommendation UI (910) according to the output type indicated by the prompt.

[0133] In one example, the generative AI model (730) may determine the output type of the recommendation UI (910) based on contextual information. The generative AI model (730) may determine the output type based on information of the user and / or information of the electronic device (10), even if the output type is not indicated by a prompt. For example, if the moving speed of the electronic device (10) is greater than a threshold or the electronic device (10) is not at a designated location (e.g., home, work, or school), the generative AI model (730) may cause the recommendation UI (910) to be output (e.g., delayed output) when the display of the execution screen of the currently running application is terminated.

[0134] FIG. 9d illustrates a recommendation UI of an electronic device according to one embodiment.

[0135] Referring to FIGS. 1 to 8 and FIG. 9D , according to one embodiment, the electronic device (10) may display a recommendation UI (960). The fourth screen (904) of FIG. 9D may correspond to a screen (e.g., a home screen of the electronic device (10)) displayed after the display of the message screen (501) of FIG. 5 is terminated. In the example of FIG. 9D , the recommendation UI (960) may include a link (e.g., a deep link within the application) of an M application that allows viewing of the movie AAA.

[0136] Although various types of recommendation UIs (960) have been described with reference to FIGS. 9A through 9D , the embodiments of the present disclosure are not limited thereto. For example, the electronic device (10) may provide recommendations in the form of simple links (e.g., deep links). For example, the electronic device (10) may register recommended actions as a schedule and provide a recommendation UI as a reminder of the registered schedule. For example, the electronic device (10) may provide information about recommended actions through notification messages (e.g., system notifications).

[0137] With reference to FIGS. 9A to 9D , examples based on the message conversation of FIG. 5 have been described, but embodiments of the present disclosure are not limited thereto. According to one embodiment, the electronic device (10) can identify a recommended action from a conversation associated with a call (e.g., a video call or a voice call). For example, the electronic device (10) can perform automatic speech recognition on voices transmitted and received through the call. The electronic device (10) can identify a recommended action based on text data acquired through the automatic speech recognition. For example, the electronic device (10) can identify a recommended action according to the method described above with reference to FIGS. 5 to 8 . If provision of a recommendation UI is determined, the electronic device (10) can provide a recommendation UI for the recommended action after the call is terminated. For example, the electronic device (10) can provide a recommendation UI according to the examples described above with reference to FIGS. 9C and 9D .

[0138] In the examples of FIGS. 9A to 9D , the electronic device (10) is described as determining a single output format, but embodiments of the present disclosure are not limited thereto. For example, the electronic device (10) may be configured to provide a recommendation UI while changing the output format. The electronic device (10) may be configured to first provide the recommendation UI of FIG. 9A or 9B , and then, when the display of the execution screen of the corresponding application (e.g., a message application) is terminated, provide the recommendation UI of FIG. 9C or 9D .

[0139] FIG. 10 illustrates additional information UIs of an electronic device according to one embodiment.

[0140] Referring to FIGS. 1 to 8 and 10, according to one embodiment, the electronic device (10) may output at least one additional information UI for obtaining additional information. The additional information UI is a UI corresponding to a recommended action of obtaining additional information, and may also be referred to as a recommendation UI.

[0141] The electronic device (10) may output a first additional information UI (1010). For example, the electronic device (10) may generate the first additional information UI (1010) using a prompt generated according to operation 810 of FIG. 8. The first additional information UI (1010) may include, for example, guide information (1011) guiding the installation of an uninstalled application and an installation link (1013) for the application. When an input for the installation link (1013) is received, the electronic device (10) may display a screen (e.g., a store page) for installing the application.

[0142] The electronic device (10) may output a second additional information UI (1020). For example, the electronic device (10) may generate the second additional information UI (1020) using a prompt generated according to operation 820 of FIG. 8. The second additional information UI (1020) may include, for example, guide information (1021) guiding application login, a login image (1022), an ID input window (1023), and a password input window (1024).

[0143] FIG. 11 illustrates a recommendation UI of an electronic device according to one embodiment.

[0144] Referring to FIGS. 1, 3, and 11, the message application of the electronic device (10) may display a message screen (1101) on the display (160). For example, a first received message (1111) may include an inquiry about the release of a movie (e.g., AAA). A first sent message (1121) may include information indicating that the movie has not yet been released. A second received message (1112) may be a message received after a certain period of time has elapsed since the first sent message (1121) was sent. The second received message (1112) may include a suggestion for watching a movie. The second sent message (1122) may include an intention to agree to watching a movie.

[0145] In one example, the electronic device (10) may generate a recommendation UI (1130) using temporally non-sequential data. For example, the electronic device (10) may determine to generate the recommendation UI (1130) using the trigger management module (320). The trigger management module (320) may determine to generate the recommendation UI (1130) based on a suggestion (e.g., how about) in the second received message (1112) and an agreement (e.g., OK) in the second sent message (1122).

[0146] In the example of FIG. 11, the second received message (1112) and the second sent message (1122) may not include information about the movie to be viewed. In one example, the UI generation module (330) may obtain information (e.g., AAA) about the movie to be viewed from the first received message (1111) stored in the database (390). For example, the electronic device (10) may obtain additional information from a conversation history with the same party.

[0147] The electronic device (10) can display a recommendation UI (1130) generated by the UI generation module (330). For example, the recommendation UI (1130) can include a thumbnail (1145) and an application link (1140). When an input for the thumbnail (1145) is received, the electronic device (10) can play a trailer for a recommended movie. When an input for the application link (1140) is received, the electronic device (10) can execute an application (e.g., T application) for reserving a recommended movie.

[0148] In one example, when an input for an application link (1140) is received, the electronic device (10) may provide a screen that allows the user to reserve a ticket for the movie AAA at a specific movie theater (e.g., a movie theater adjacent to the electronic device (10)) at a specific time (e.g., a time after the current time). After a certain period of time has elapsed since the recommendation UI (1130) is provided, the link (e.g., a deep link) of the recommendation UI (1130) may need to be changed. For example, the available reservation times and / or available movie theaters may change as time elapses. For example, the electronic device (10) may monitor information on available reservation times and movie theaters using an artificial intelligence model. If the information on available reservation times and / or movie theaters changes, the electronic device (10) may generate a new recommendation UI based on the changed information. As another example, after a certain period of time has elapsed since the recommendation UI (1130) was provided (e.g., without executing a recommendation action), the electronic device (10) may generate a new recommendation UI.

[0149] While examples related to multimedia have been described with reference to FIGS. 5 through 11, embodiments of the present disclosure are not limited thereto. For example, the electronic device (10) may identify financial transactions related to remittance, deposit, and / or withdrawal based on the user's intent. In this case, the electronic device (10) may provide a recommendation UI that includes information for performing the financial transaction (e.g., a link to execute a banking application).

[0150] Fig. 12 is a flowchart of a method for providing a UI of an electronic device according to one embodiment.

[0151] The operations described below with reference to FIG. 12 may be referred to as operations of the electronic device (10) of FIG. 1. The order of the operations described below with reference to FIG. 12 is merely an example, and embodiments of the present disclosure are not limited thereto. For example, at least some of the operations may be executed differently from the order of FIG. 12, or may be executed substantially simultaneously with other operations of FIG. 12. At least some of the operations described below with reference to FIG. 12 may be omitted.

[0152] Referring to FIGS. 1 to 8 and FIG. 12 , in operation 1205, the electronic device (10) may identify data-based keywords and intents. For example, the electronic device (10) may identify keywords and intents from data collected by the data management module (310) of FIG. 3 . The data may be data obtained from an application currently running on the electronic device (10).

[0153] For example, the electronic device (10) can identify keywords and intents from data using an AI model. The AI ​​model may include an LLM configured to extract keywords from collected data. The electronic device (10) can use the AI ​​model to identify at least one keyword from the data and an intent corresponding to the at least one keyword.

[0154] In operation 1210, the electronic device (10) may determine whether to recommend an action. The electronic device (10) may determine whether to recommend an action based on identified keywords and / or intent. For example, the electronic device (10) may determine whether to recommend an action using the trigger management module (320) of FIG. 3. For example, if the trigger management module (320) determines to generate a recommendation UI, the electronic device (10) may determine to recommend an action. If the trigger management module (320) determines not to generate a recommendation UI, the electronic device (10) may determine not to recommend an action. If it is determined not to recommend an action (e.g., operation 1210-NO), the electronic device (10) may terminate the execution of the UI providing method without performing any additional actions.

[0155] For example, the electronic device (10) may positively determine whether to recommend an action if a keyword indicating the user's intention and / or willingness is identified from the identified keywords and / or intentions. For example, the electronic device (10) may determine whether to recommend an action according to the method described above with respect to the trigger resolver (630) described above with respect to FIG. 5 .

[0156] If an action recommendation is determined (e.g., action 1210-YES), in action 1215, the electronic device (10) may identify a recommended action. For example, the electronic device (10) may identify a recommended action based on the identified keywords and / or intents. The electronic device (10) may obtain additional information associated with the identified keywords and / or intents and identify a recommended action further based on the additional information. In one example, the recommended action may include generating a recommendation UI for recommending the target action. The electronic device (10) may identify a recommended action according to the actions described above with respect to the data analyzer (620) of FIG. 6 .

[0157] In operation 1220, the electronic device (10) may determine whether installation of an application is required. For example, the electronic device (10) may determine whether an application associated with the identified recommended action (e.g., an application required to perform the purpose of the recommended action) is installed on the electronic device (10). The electronic device (10) may determine whether to install the application based on data obtained from a database (e.g., database (390) of FIG. 3), an OS, and / or an application. If the corresponding application is installed on the electronic device (10), the electronic device (10) may determine that installation of the application is not required. The electronic device (10) may determine whether to install the application according to operation 805 of FIG. 8.

[0158] If application installation is required (e.g., operation 1220-YES), in operation 1225, the electronic device (10) may generate an application installation UI and provide the generated UI. For example, the electronic device (10) may generate and provide a UI that guides application installation. The electronic device (10) may generate a prompt for generating the installation UI. The electronic device (10) may generate the prompt according to, for example, operation 810 of FIG. 8.

[0159] With respect to operations 1220 and 1225, provision of a UI based on whether an application is installed has been described. In one example, the electronic device (10) may determine whether a function associated with an identified recommended action (e.g., a function required to perform the purpose of the recommended action) is activated. For example, the function may be deactivated on the electronic device (10) or may require additional permission settings to execute the function. In this case, the electronic device (10) may provide a UI for activating the function. The UI may include guide information for activating the function. If the function is activated, the electronic device (10) may generate a prompt according to operation 1230.

[0160] If application installation is not required (e.g., operation 1220-NO), at operation 1230, the electronic device (10) may generate a prompt for generating a UI associated with the recommended action. The electronic device (10) may generate at least one prompt based on keywords, intents, and / or additional information. For example, the electronic device (10) may generate a prompt according to the operation described above with respect to the prompt generation module (720) of FIG. 7. As described above with respect to FIG. 7, the electronic device (10) may generate multiple prompts. In one example, the electronic device (10) may generate a prompt according to operation 830 or 835 of FIG. 8.

[0161] In operation 1235, the electronic device (10) can generate and provide a UI based on the prompt. The electronic device (10) can generate a UI (e.g., a recommended UI) by inputting the generated prompt into a generative AI model (e.g., the generative AI model (730) of FIG. 7). For example, the electronic device (10) can generate a UI using the UI generation module (330) of FIG. 3.

[0162] According to one embodiment, the electronic device (10) may include a rollable display (e.g., display (160)). The provision of a recommendation UI may be determined when the electronic device (10) is in a retracted state. In this case, the electronic device (10) may expand the size of the visible area of ​​the display by rolling out the rollable display and provide the recommendation UI on the rolled-out display. In one example, the electronic device (10) may provide guide information for rolling out the display (160) to provide the recommendation UI.

[0163] According to one embodiment, the electronic device (10) may include a foldable display (e.g., display (160)). Provision of a recommendation UI may be determined in a folded state of the electronic device (10). In one example, the electronic device (10) may provide the recommendation UI when the electronic device (10) is unfolded. The electronic device (10) may provide guide information for unfolding the electronic device (10) through a display area that is visible from the outside in the folded state (e.g., a display separate from the foldable display or a portion of the foldable display).

[0164] According to one embodiment, the electronic device (10) may provide a recommendation UI through an external electronic device. For example, the electronic device (10) may transmit information about the recommendation UI to the external electronic device using a communication circuit (190). The external electronic device may then provide the information using the received information about the recommendation UI. The external electronic device may be, for example, a device configured to support continuity with the electronic device (10).

[0165] Fig. 13 is a flowchart of a method for obtaining a recommendation UI of an electronic device according to one embodiment.

[0166] The operations described below with reference to FIG. 13 may be referred to as operations of the electronic device (10) of FIG. 1. The order of the operations described below with reference to FIG. 13 is merely an example, and embodiments of the present disclosure are not limited thereto. For example, at least some of the operations may be executed differently from the order of FIG. 13, or may be executed substantially simultaneously with other operations of FIG. 13. At least some of the operations described below with reference to FIG. 13 may be omitted.

[0167] Referring to FIGS. 1 to 12 and 13, according to one embodiment, an electronic device (10) may include a display (160), at least one processor (120) electrically connected to the display (160), and a memory (130) electrically connected to the at least one processor (120) and storing instructions. The instructions, when executed by the at least one processor (120), may cause the electronic device (10) to perform operations described below.

[0168] In operation 1305, the electronic device (10) can identify keywords and / or intentions from data. For example, the electronic device (10) can identify at least one keyword and / or at least one intention from data (e.g., images and / or text) including at least one of a message, schedule information, or input information using a first artificial intelligence model. The electronic device (10) can extract keywords and / or intentions from the data by inputting the data into the first artificial intelligence model (LLM). For example, the electronic device (10) can identify keywords and / or intentions according to operation 1205 of FIG. 12 .

[0169] For example, the electronic device (10) can store structured data in the memory (130). The electronic device (10) can store data according to the operation of the data management module (310) of FIG. 3. The electronic device (10) can process the structured data using the first artificial intelligence model to identify keywords and / or intent.

[0170] In operation 1310, the electronic device (10) may identify a recommended action based on at least one of the identified keywords or intents. For example, the electronic device (10) may identify a recommended action based on the identified keywords and / or intents. The electronic device (10) may obtain additional information associated with the identified keywords and / or intents and identify a recommended action further based on the additional information. In one example, the recommended action may include generating a recommendation UI for recommending a target action.

[0171] In one example, the electronic device (10) may determine whether to recommend an action based on the identified keyword. For example, the electronic device (10) may determine whether to recommend an action according to operation 1210 of FIG. 12. By determining to recommend an action, the electronic device (10) may identify the recommended action.

[0172] In operation 1315, the electronic device (10) may generate at least one prompt for generating a recommendation UI using a second artificial intelligence model. In one example, the second artificial intelligence model may be a generative AI model (e.g., the generative AI model (250) of FIG. 2 and / or the generative AI model (730) of FIG. 7). For example, the electronic device (10) may generate at least one prompt by inputting information about a recommended action into the second artificial intelligence model. The electronic device (10) may generate at least one prompt according to operation 1230 of FIG. 12.

[0173] In one example, at least one prompt may include a first prompt and a second prompt. The first prompt may include information indicating the structure of the recommendation UI. The structure of the recommendation UI may include information about the color, size, display location, components, and / or related actions (e.g., intents or deep links) of the recommendation UI. The second prompt may include information indicating resources of the recommendation UI. For example, the electronic device (10) may obtain the structure of the recommendation UI by inputting the first prompt into a generative AI model (e.g., the first generative AI model). The electronic device (10) may obtain UI resources by inputting the second prompt into a generative AI model (e.g., the second generative AI model). The electronic device (10) may generate the recommendation UI by combining the UI resources with the structure of the UI.

[0174] In operation 1320, the electronic device (10) may obtain a recommendation UI including guide information for a recommended action. For example, the electronic device (10) may obtain the recommendation UI by inputting at least one prompt to a third generative artificial intelligence model (e.g., the generative AI model (250) of FIG. 2 and / or the generative AI model (730) of FIG. 7). For example, the electronic device (10) may obtain the recommendation UI according to operation 1230 of FIG. 12. For example, the recommendation UI may include at least one of a widget, a floating window, a graphic object, an image, an icon, or schedule information.

[0175] In one example, a recommended action may be associated with a specified application or a specified function. If the specified application is not installed on the electronic device (10), the electronic device (10) may generate a recommendation UI that includes guide information for installing the specified application, and if the specified function is disabled, the recommendation UI may generate a recommendation UI that includes guide information for activating the specified function.

[0176] In operation 1325, the electronic device (10) may display a recommendation UI. The electronic device (10) may display the recommendation UI on the display (160). The electronic device (10) may provide the recommendation UI as a widget, an overlay image, or an in-application message. As described above with respect to FIGS. 9A to 9D , the electronic device (10) may display the recommendation UI based on context information. For example, the electronic device (10) may provide the recommendation UI after the display of the execution screen of the currently executing application is terminated. As another example, the electronic device (10) may display the recommendation UI on the execution screen of the currently executing application. The context information may include, for example, the location of the electronic device (10), and the location of the electronic device may include at least one of home, work, or within a vehicle identified based on the moving speed of the electronic device.

[0177] FIG. 14 is a block diagram of an electronic device (1401) within a network environment (1400) according to various embodiments. Referring to FIG. 14, in the network environment (1400), the electronic device (1401) may communicate with the electronic device (1402) via a first network (1498) (e.g., a short-range wireless communication network), or may communicate with at least one of the electronic device (1404) or the server (1408) via a second network (1499) (e.g., a long-range wireless communication network). In one embodiment, the electronic device (1401) may communicate with the electronic device (1404) via the server (1408). According to one embodiment, the electronic device (1401) may include a processor (1420), a memory (1430), an input module (1450), an audio output module (1455), a display module (1460), an audio module (1470), a sensor module (1476), an interface (1477), a connection terminal (1478), a haptic module (1479), a camera module (1480), a power management module (1488), a battery (1489), a communication module (1490), a subscriber identification module (1496), or an antenna module (1497). In some embodiments, the electronic device (1401) may omit at least one of these components (e.g., the connection terminal (1478)), or may have one or more other components added. In some embodiments, some of these components (e.g., sensor module (1476), camera module (1480), or antenna module (1497)) may be integrated into a single component (e.g., display module (1460)).

[0178] The processor (1420) may, for example, execute software (e.g., a program (1440)) to control at least one other component (e.g., a hardware or software component) of the electronic device (1401) connected to the processor (1420) and perform various data processing or operations. According to one embodiment, as at least a part of the data processing or operations, the processor (1420) may store commands or data received from other components (e.g., a sensor module (1476) or a communication module (1490)) in a volatile memory (1432), process the commands or data stored in the volatile memory (1432), and store result data in a non-volatile memory (1434). According to one embodiment, the processor (1420) may include a main processor (1421) (e.g., a central processing unit or an application processor) or an auxiliary processor (1423) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together with the main processor (1421). For example, when the electronic device (1401) includes the main processor (1421) and the auxiliary processor (1423), the auxiliary processor (1423) may be configured to use less power than the main processor (1421) or to be specialized for a given function. The auxiliary processor (1423) may be implemented separately from the main processor (1421) or as a part thereof.

[0179] The auxiliary processor (1423) may control at least a portion of functions or states associated with at least one component (e.g., the display module (1460), the sensor module (1476), or the communication module (1490)) of the electronic device (1401), for example, on behalf of the main processor (1421) while the main processor (1421) is in an inactive (e.g., sleep) state, or together with the main processor (1421) while the main processor (1421) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (1423) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (1480) or a communication module (1490)). In one embodiment, the auxiliary processor (1423) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (1401) where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (1408)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0180] The memory (1430) can store various data used by at least one component (e.g., the processor (1420) or the sensor module (1476)) of the electronic device (1401). The data can include, for example, software (e.g., the program (1440)) and input data or output data for commands related thereto. The memory (1430) can include volatile memory (1432) or non-volatile memory (1434).

[0181] The program (1440) may be stored as software in memory (1430) and may include, for example, an operating system (1442), middleware (1444), or an application (1446).

[0182] The input module (1450) can receive commands or data to be used in a component of the electronic device (1401) (e.g., a processor (1420)) from an external source (e.g., a user) of the electronic device (1401). The input module (1450) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0183] The audio output module (1455) can output audio signals to the outside of the electronic device (1401). The audio output module (1455) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0184] The display module (1460) can visually provide information to an external party (e.g., a user) of the electronic device (1401). The display module (1460) may include, for example, a display, a holographic device, or a projector, and a control circuit for controlling the device. In one embodiment, the display module (1460) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0185] The audio module (1470) can convert sound into an electrical signal, or vice versa. According to one embodiment, the audio module (1470) can acquire sound through the input module (1450), output sound through the sound output module (1455), or an external electronic device (e.g., electronic device (1402)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (1401).

[0186] The sensor module (1476) can detect the operating status (e.g., power or temperature) of the electronic device (1401) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (1476) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0187] The interface (1477) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (1401) with an external electronic device (e.g., the electronic device (1402)). In one embodiment, the interface (1477) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0188] The connection terminal (1478) may include a connector through which the electronic device (1401) may be physically connected to an external electronic device (e.g., the electronic device (1402)). In one embodiment, the connection terminal (1478) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0189] The haptic module (1479) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. In one embodiment, the haptic module (1479) may include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0190] The camera module (1480) can capture still images and videos. In one embodiment, the camera module (1480) may include one or more lenses, image sensors, image signal processors, or flashes.

[0191] The power management module (1488) can manage the power supplied to the electronic device (1401). According to one embodiment, the power management module (1488) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0192] A battery (1489) may power at least one component of the electronic device (1401). In one embodiment, the battery (1489) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0193] The communication module (1490) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (1401) and an external electronic device (e.g., electronic device (1402), electronic device (1404), or server (1408)), and the performance of communication through the established communication channel. The communication module (1490) may operate independently from the processor (1420) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (1490) may include a wireless communication module (1492) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (1494) (e.g., a local area network (LAN) communication module, or a power line communication module). Any of these communication modules may communicate with an external electronic device (1404) via a first network (1498) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (1499) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a local area network or a wide area network)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (1492) may use subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (1496) to verify or authenticate the electronic device (1401) within a communication network such as the first network (1498) or the second network (1499).

[0194] The wireless communication module (1492) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimizing terminal power and connecting multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency communications (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (1492) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (1492) may support various technologies for securing performance in high-frequency bands, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (1492) may support various requirements specified in the electronic device (1401), an external electronic device (e.g., the electronic device (1404)), or a network system (e.g., the second network (1499)). According to one embodiment, the wireless communication module (1492) may support a peak data rate (e.g., 20 Gbps or more) for eMBB implementation, a loss coverage (e.g., 164 dB or less) for mMTC implementation, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC implementation.

[0195] The antenna module (1497) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (1497) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (1497) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (1498) or the second network (1499), may be selected from the plurality of antennas by, for example, the communication module (1490). A signal or power may be transmitted or received between the communication module (1490) and the external electronic device via the selected at least one antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (1497).

[0196] According to various embodiments, the antenna module (1497) may form a mmWave antenna module. In one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high frequency band.

[0197] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0198] According to one embodiment, commands or data may be transmitted or received between the electronic device (1401) and an external electronic device (1404) via a server (1408) connected to a second network (1499). Each of the external electronic devices (1402 or 1404) may be the same or a different type of device as the electronic device (1401). According to one embodiment, all or part of the operations executed in the electronic device (1401) may be executed in one or more of the external electronic devices (1402, 1404, or 1408). For example, when the electronic device (1401) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (1401) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (1401). The electronic device (1401) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (1401) may provide an ultra-low latency service using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device (1404) may include an Internet of Things (IoT) device. The server (1408) may be an intelligent server utilizing machine learning and / or a neural network.According to one embodiment, an external electronic device (1404) or server (1408) may be included within the second network (1499). The electronic device (1401) may be applied to intelligent services (e.g., smart homes, smart cities, smart cars, or healthcare) based on 5G communication technology and IoT-related technology.

[0199] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

[0200] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0201] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0202] Various embodiments of the present document may be implemented as software (e.g., a program (1440)) including one or more instructions stored in a storage medium (e.g., an internal memory (1436) or an external memory (1438)) readable by a machine (e.g., an electronic device (1401)). For example, a processor (e.g., a processor (1420)) of the machine (e.g., an electronic device (1401)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0203] According to one embodiment, the method according to various embodiments disclosed in this document may be provided as included in a computer program product. The computer program product may be traded as a commodity between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play Store™) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0204] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In an electronic device (10, 1401), display(160, 1460); At least one processor (120, 1420) electrically connected to the display; and A memory (130, 1430) electrically connected to at least one processor and storing instructions, When the above instructions are executed by the at least one processor, the electronic device: Using a first artificial intelligence model, at least one of one or more keywords or one or more intents is identified from data including at least one of message information, schedule information, or input information, Identifying a recommended action based on at least one of the above one or more keywords or one or more of the above one or more intents, Generate at least one prompt for generating a recommendation UI using the second artificial intelligence model, By inputting at least one of the above prompts into a third generative artificial intelligence model, the recommendation UI including guide information for the above recommended action is obtained, An electronic device that displays the above recommended UI on the display.

2. In paragraph 1, When the above instructions are executed by the at least one processor, the electronic device: An electronic device that obtains one or more keywords or one or more intents by inputting at least one of the images or texts included in the data into the first artificial intelligence model.

3. In paragraph 2, When the above instructions are executed by the at least one processor, the electronic device: Decide whether to recommend an action based on one or more of the above keywords, An electronic device that identifies said recommended action based on which the device determines to recommend said action.

4. In paragraph 2, When the above instructions are executed by the at least one processor, the electronic device: An electronic device that generates at least one prompt by inputting information of the above recommended action into the second artificial intelligence model.

5. In paragraph 1, When the above instructions are executed by the at least one processor, the electronic device: Store at least one of the above message information, the above schedule information, or the above input information as structured data in the memory, An electronic device that processes the structured data using the first artificial intelligence model to identify at least one of the one or more keywords or the one or more intents.

6. In paragraph 1, The above recommended actions are associated with a specified application or a specified function, If the above-mentioned specified application is not installed on the electronic device, the recommendation UI includes guide information for installation of the above-mentioned specified application, An electronic device, wherein the recommendation UI includes guide information for activating the specified function when the specified function is disabled.

7. In paragraph 1, When the above instructions are executed by the at least one processor, the electronic device: An electronic device that provides the above recommended UI as a widget, overlay image, or in-application message.

8. In paragraph 7, When the above instructions are executed by the at least one processor, the electronic device, based on context information: The above recommended UI is provided after the display of the execution screen of the currently running application is terminated, or An electronic device that provides the above recommended UI on the execution screen of the currently running application.

9. In paragraph 8, The context information includes the location of the electronic device, An electronic device, wherein the location of the electronic device comprises at least one of a home, a business, or a vehicle identified based on a moving speed of the electronic device.

10. In paragraph 1, An electronic device, wherein the above recommended UI includes at least one of a widget, a floating window, a graphic object, an image, an icon, a message, or a schedule.

11. A method for providing a recommended UI for an electronic device (10, 1401), An operation of identifying at least one of one or more keywords or one or more intents from data including at least one of message information, schedule information, or input information using a first artificial intelligence model; An action that identifies a recommended action based on at least one of said one or more keywords or said one or more intents; An action of generating at least one prompt for generating a recommendation UI using a second artificial intelligence model; An operation of obtaining the recommendation UI including guide information for the recommended action by inputting at least one of the above prompts into a third generative artificial intelligence model; and A method comprising an action of displaying the above recommended UI.

12. In paragraph 11, An action that identifies at least one of the above one or more keywords or one or more intents, A method comprising an action of obtaining said one or more keywords or said one or more intents by inputting at least one of the images or texts included in said data into said first artificial intelligence model.

13. In paragraph 12, Further comprising an action for determining whether to recommend an action based on one or more of the above keywords, A method wherein the action of identifying the above recommended action is performed based on a decision to recommend the above action.

14. In paragraph 12, A method wherein the action of generating said at least one prompt comprises an action of generating said at least one prompt by inputting information of said recommended action into said second artificial intelligence model.

15. In paragraph 11, Further comprising an action of storing at least one of the message information, the schedule information, or the input information as structured data in a memory of the electronic device, A method wherein the act of identifying at least one of the one or more keywords or the one or more intents comprises an act of identifying at least one of the one or more keywords or the one or more intents by processing the structured data using the first artificial intelligence model.

Images