Method and apparatus for generating response from query using generative model

Abstract

The electronic device comprises: a display; at least one processor comprising processing circuitry; and a memory for storing instructions, wherein the at least one processor is configured to execute the instructions individually and / or collectively, and cause the electronic device to: extract a plurality of pieces of candidate data related to a query from among source data on the basis of obtaining the query for the source data; select input data from among the extracted plurality of pieces of candidate data on the basis of contents of the plurality of pieces of candidate data; generate a response to the query by applying the query and the selected input data to a generative model; determine a partial response derived from the selected input data from the response; and display, through the display, a visual representation indicating information about the selected input data in a region corresponding to the determined partial response.

Description

Method and device for generating a response from a query using a generative model

[0001] The present disclosure relates to a technique for generating a response from a query using a generative model.

[0002] Recently, techniques for generating responses from queries using generative models have advanced significantly. However, responses generated by generative models are not necessarily reliable and can contain errors known as hallucinations.

[0003] The above information may be provided as background information to aid in understanding this document. None of the above is claimed to be prior art related to this document or can be used to determine prior art.

[0004] An electronic device comprises: a display; at least one processor including a processing circuit; and a memory storing instructions, wherein the at least one processor is configured to individually and / or collectively execute the instructions, and cause the electronic device to: obtain a query for querying source data, extract a plurality of pieces of candidate data related to the query from the source data; select input data from among the plurality of extracted candidate data based on contents of the plurality of candidate data; generate a response to the query by applying the query and the selected input data to a generative model; determine a partial response derived from the selected input data among the responses; and display, through the display, a visual representation indicating information about the selected input data in an area corresponding to the determined partial response.

[0005] A method performed by an electronic device may include: obtaining a query for querying source data; extracting a plurality of pieces of candidate data related to the query from among the source data; selecting input data from among the plurality of extracted candidate data based on contents of the plurality of candidate data; generating a response to the query by applying the query and the selected input data to a generative model; determining a partial response derived from the selected input data among the responses; and displaying a visual representation indicating information about the selected input data in an area corresponding to the determined partial response through a display.

[0006] The foregoing and other aspects, features, and advantages of specific embodiments of the present disclosure will become more apparent from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0007] FIG. 1 is a block diagram illustrating an exemplary configuration of an electronic device within a network environment according to various embodiments.

[0008] FIG. 2 is a diagram illustrating an example of an artificial intelligence system according to various embodiments.

[0009] FIG. 3 is a flowchart illustrating an example of how an electronic device generates a response from a query according to various embodiments.

[0010] FIG. 4 illustrates an example of an interface for an electronic device to obtain a query and output a response according to various embodiments.

[0011] FIG. 5 is a diagram illustrating an example of an operation of an electronic device according to various embodiments to select input data from among multiple candidate data having the same content.

[0012] FIG. 6 is a diagram illustrating an example of an operation of an electronic device according to various embodiments to select input data from among a plurality of candidate data having different contents.

[0013] FIG. 7A illustrates an example of a response to a query generated by an electronic device using specified source data according to various embodiments.

[0014] FIG. 7b illustrates an example of a response to a query generated by an electronic device when source data is not specified, according to various embodiments.

[0015] FIG. 8 is a diagram illustrating an example of an operation of an electronic device according to various embodiments to display a visual representation indicating information about a generative model used to generate a response.

[0016] FIG. 9 illustrates an example of an interface for obtaining a query and outputting a response when an electronic device designates user-provided data and search data as source data according to various embodiments.

[0017] FIG. 10 illustrates an example of an interface for obtaining a query and outputting a response when an electronic device, according to various embodiments, specifies search data as source data rather than user-provided data as source data.

[0018] FIG. 11 illustrates an example of a screen for specifying data related to a specific application as source data according to various embodiments.

[0019] FIG. 12 illustrates an example of a screen for specifying source data based on a directory in an electronic device according to various embodiments.

[0020] FIG. 13 is a diagram illustrating examples of screens that include responses of electronic devices according to various embodiments.

[0021] FIG. 14 is a diagram illustrating an example configuration for performing operations of an electronic device according to various embodiments.

[0022] Hereinafter, embodiments will be described in detail with reference to the attached drawings. In the description with reference to the attached drawings, identical components are assigned the same reference numerals regardless of the drawing numbers, and redundant descriptions thereof will be omitted.

[0023] FIG. 1 is a block diagram illustrating an exemplary configuration of an electronic device within a network environment according to various embodiments.

[0024] FIG. 1 is a block diagram of an electronic device (101) within a network environment (100) according to various embodiments. Referring to FIG. 1, in the network environment (100), the electronic device (101) may communicate with an electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of an electronic device (104) or a server (108) via a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108).

[0025] According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display (160), an audio module (170), a sensor (176), an interface (177), a connection terminal (178), a haptic module (179), a camera (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In various embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In various embodiments, some of these components (e.g., the sensor (176), the camera (180), or the antenna module (197)) may be integrated into one component (e.g., the display (160)).

[0026] The processor (120) may be implemented as one or more IC (integrated circuit (or circuitry)) chips and may perform various data processing. The processor (120) may include at least one electrical circuit and may individually or collectively perform distributed processing of instructions (or programs (140), data, etc.) stored in the memory (130). The processor (120) may include a processor assembly including one or more processing circuits. The processor (120) may include any processing circuit that is operative to control the performance and operations of one or more components (e.g., the memory (130), the display (160), the camera (180), the communication module (190), and / or the sensor (176)) of the electronic device (101). The processor (120) may execute various instructions to provide a model (e.g., an artificial intelligence model (AI model)) and / or an AI framework according to various embodiments.

[0027] The processor (120) may include various processing circuits and / or multiple processors. For example, as used herein, including in the claims, the term "processor" may include various processing circuits, including at least one processor, one or more of which may be configured to perform the various functions described herein, collectively and / or individually, in a distributed manner. As used herein, when "processor," "at least one processor," and "one or more processors" are described as being configured to perform numerous functions, these terms may cover situations, such as, but not limited to, a situation where one processor performs some of the listed functions, other processor(s) perform other of the listed functions, and a situation where a single processor performs all of the listed functions. Additionally, the at least one processor may include a combination of processors that perform the various listed / described functions, e.g., in a distributed manner. At least one processor may execute program instructions to perform or achieve various functions. The processor (120) may execute various instructions to provide an AI framework and / or machine training and / or learning according to various embodiments, and / or to invoke a model (e.g., an artificial intelligence model). Here, any model (e.g., an AI model) may include a processor including a processing circuit.

[0028] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or calculations. According to one embodiment, as at least a part of the data processing or calculations, the processor (120) may store commands or data received from other components (e.g., a sensor (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or a secondary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor)) that can operate independently or together therewith. For example, if the electronic device (101) includes a main processor (121) and a secondary processor (123), the secondary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a specified function. The secondary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0029] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display (160), a sensor (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0030] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0031] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0032] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0033] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0034] A display (160) (e.g., a display) can visually provide information to an external device (e.g., a user) of the electronic device (101). The display (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. In one embodiment, the display (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0035] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0036] The sensor (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor (176) may include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an infrared (IR) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor. For example, the sensor (176) may include an inertial measurement unit (IMU).

[0037] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0038] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0039] A haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. In one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0040] The camera (180) can capture still images and moving images. According to one embodiment, the camera (180) can include one or more lenses, one or more image sensors, one or more image signal processors, or one or more flashes.

[0041] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least a part of a power management integrated circuit (PMIC).

[0042] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0043] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may include one or more communication circuits. The communication module (190) may include one or more communication processors (CPs) that operate independently from the processor (120) (e.g., application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Any of these communication modules may communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data relation association)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as a plurality of separate components (e.g., multiple chips).The wireless communication module (192) can identify or authenticate an electronic device (101) within a communication network, such as a first network (198) or a second network (199), using subscriber information (e.g., international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0044] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0045] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). According to one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas by, for example, the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. According to various embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0046] According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

[0047] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0048] According to one embodiment, commands or data may be transmitted or received between an electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199).

[0049] Each of the external electronic devices (102, 104) and the server (108) may be the same type of device as or different from the electronic device (101). According to one embodiment, all or part of the operations executed by the electronic device (101) may be executed by one or more of the external electronic devices (102, 104) or the server (108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of executing the function or service itself or in addition, request one or more external electronic devices to execute the function or at least a part of the service. The one or more external electronic devices that receive the request may execute at least a part of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a part of a response to the request.

[0050] FIG. 2 is a diagram illustrating an example of an artificial intelligence system according to various embodiments.

[0051] In an artificial intelligence system (hereinafter, “system”) (200), a user query / response interface (210) can receive user input. The user input can be input in the form of natural language, images, audio, and / or video. The user input can be transmitted together with context information. The context information can include various side information related to the time at which the user input is input into the system (200). For example, the context information can include information about the application currently being used by the user or information about the user’s location. In addition, the user input can be a mixed type of input of two or more of the above-described natural language, images, audio, video, and / or context information. In addition, the user input can include non-natural language input, such as selecting a menu.

[0052] The user query / response interface (210) can provide the user with output from the generative artificial intelligence system. The output may include a natural language-based response and / or specific content. The output may also include an action requested by the user.

[0053] An artificial intelligence framework (AI framework) (220) can receive user input. Based on the user input (e.g., the user's query), the AI ​​framework (220) can coordinate and control one or more components necessary to perform an action corresponding to the user's intent.

[0054] User input received from the user query / response interface (210) may be transmitted to a prompt design component (221). The prompt design component (221) may be used to generate a prompt suitable as input to a generative model (e.g., a large language model (LLM) and / or a large multimodal model (LMM)) based on the user input.

[0055] The prompt design component (221) may include various circuits and / or executable program instructions and may be an AI component using a machine learning algorithm or a neural network. The prompt design component (221) may generate improved prompts through learning over time. The prompt design component (221) may access a knowledge storage (230) to generate prompts based on user input. The knowledge storage (230) may store user preference data, a prompt library, and / or prompt examples. The prompt design component (223) may provide the generated prompts to a generative model (250) (e.g., an LLM and / or an LMM).

[0056] The APIs / Plugins management component (223) may include various circuits and / or executable program instructions and may communicate with external information sources based on requests for additional information when user input is transmitted to the generative model (250).

[0057] The APIs / Plugins management component (223) may include various circuits and / or executable program instructions and may establish a communication channel for communication with the outside of the system (200) via the APIs. The APIs / Plugins management component (223) may enable the AI ​​framework (220) to access various data sources via the communication channel. Information obtained from the data sources may be used to generate prompts by the prompt design component (221) together with user input or may be used as input to the generative model (250).

[0058] The APIs / Plugins management component (223) can request a final action via an API when a final action in response to user input, rather than an intermediate action, must be performed by an application or service.

[0059] The refiner component (225) can fine-tune the output of the generative model (250). For example, the refiner component (225) can determine the relevance level (e.g., score) between the output (e.g., content) of the generative model (250) and the user input. For example, the refiner component (225) can determine whether the output contains biased information (e.g., selective information). For example, the refiner component (225) can determine whether the output contains harmful information (e.g., violent content and / or profanity).

[0060] The refinement component (225) can determine the degree of matching (e.g., score) between the output of the generative model (250) and the user input (e.g., the intent of the user input). If the refinement component (225) determines that the output of the generative model (250) does not correspond to the user input, it can modify the output so that it corresponds to the user input.

[0061] The refinement component (225) can provide hints (e.g., hints for prompt generation) to the user so that the user can obtain information that matches the user's intention from the generative model (250).

[0062] A generative model (250) may refer to a model including an artificial intelligence neural network that generates new data (e.g., text, images, audio, and / or video) based on user input (e.g., user utterance). The generative model (250) may include an image generation model and / or a language generation model.

[0063] Image generation models may include generative adversarial networks (GANs) and / or variational autoencoders (VAEs). Examples of image generation models include diffusion-based generative models, which have the structure of VAEs and transformers.

[0064] A language generation model (e.g., ChatGPT) can be a model trained to generate statistically most appropriate output based on input. A language generation model can include an LLM. An LLM can identify various types of input, such as text, images, audio (e.g., speech), and / or video, and generate new data corresponding to the input.

[0065] In various embodiments, an electronic device (e.g., electronic device (101) of FIG. 1) may acquire new data generated through a generative model (250) using an artificial intelligence system (200). For example, the electronic device may include part or all of the artificial intelligence system (200) and may generate new data using the artificial intelligence system (200) (e.g., on-device). In another example, the artificial intelligence system (200) may be implemented as an external device of the electronic device (e.g., electronic device (102, 104) of FIG. 1, server (108)), and the electronic device may receive new data generated using the artificial intelligence system (200) external to the electronic device from the artificial intelligence system (200) (e.g., on-cloud).

[0066] FIG. 3 is a flowchart illustrating an example of how an electronic device generates a response from a query according to various embodiments.

[0067] According to an embodiment, an electronic device (e.g., the electronic device (101) of FIG. 1) may obtain a query that queries source data. The query is a query generated based on a user input and may also be expressed as a user query. The source data may refer to data that the electronic device can use to generate a response to the query. The source data may include at least one of: at least a portion of internal data stored in a memory of the electronic device (e.g., the memory (130) of FIG. 1); at least a portion of external device data stored in another electronic device connected to the electronic device (e.g., the electronic device (102) of FIG. 1, the electronic device (104) of FIG. 1); or search data obtainable through a search server.

[0068] Other electronic devices connected to the electronic device may include devices that establish direct communication with the electronic device and / or devices that establish direct communication with an intermediate device (e.g., a server) that establishes direct communication with the electronic device. For example, if the other electronic device uploads (e.g., backs up) data of the other electronic device to an intermediate device (e.g., a server) and the electronic device can access the data stored on the intermediate device (e.g., the server), the other electronic device may be determined to be connected to the electronic device via the intermediate device (e.g., the server).

[0069] In various embodiments of the present disclosure, internal data and external device data may be collectively referred to as 'user-provided data'.

[0070] An electronic device may specify source data based on user input. For example, the source data may be specified by the user based on at least one of an application, a directory (e.g., a folder) in which internal data is stored, or an external device that stores external device data. If the source data is specified through a specific application, data generated or processed (e.g., received from and / or transmitted to another electronic device) through the specific application may be specified as the source data. If the source data is specified through a specific directory, internal data stored in the specific directory may be specified as the source data. If the source data is specified through a specific external device, data stored in and / or received from the specific external device may be specified as the source data. Examples of an operation for specifying source data based on user input and / or an interface for specifying source data are described in more detail below with reference to FIGS. 4, 9, 10, 11, and 12.

[0071] In various embodiments of the present disclosure, the source data is not limited to being specified based on user input, and the electronic device may also specify the source data based on a query.

[0072] In an embodiment, the electronic device may determine that the source data is internal data stored in the memory of the electronic device based on whether the query inquires about the user's private information. If the electronic device determines by analyzing the query that the user's intent requests a response based on the internal data, the electronic device may determine that the query inquires about private information. For example, if the query "Choose a picture of a cat among the pictures I took yesterday," the electronic device may determine that the user is requesting a response based on the internal data based on the phrase "among the pictures I took yesterday." For example, if the query "Find the travel pictures my daughter sent me," the electronic device may determine that the user is requesting a response based on the internal data based on the phrase "sent by my daughter."

[0073] According to an embodiment, an electronic device may designate source data based on whether a query can be answered with internal data. For example, the electronic device may obtain summary data of internal data stored in a memory. The summary data may be data that summarizes the internal data and may include one or more topics extracted from the internal data. Based on obtaining a query, the electronic device may determine whether the query is answerable with the internal data using the summary data and the query. Based on whether the query is answerable with the internal data, the electronic device may designate source data including the internal data. Based on whether the query is not answerable with the internal data, the electronic device may designate source data including at least one of external device data or search data. If the query is not answerable with the internal data, the electronic device may designate at least one of the internal data, external device data, or search data as the source data.

[0074] The electronic device can, after specifying source data, obtain (e.g., collect) at least a portion of the specified source data.

[0075] For example, if an electronic device designates external device data as source data, the electronic device can acquire (e.g., receive) external device data mapped to the designated external device as source data from the designated external device or an intermediate device (e.g., a cloud device) connected to the designated external device.

[0076] For example, if an electronic device specifies search data as source data, it can generate a search request (e.g., a search term or a search image) based on the query. The electronic device can transmit the generated search request to an external device (e.g., a search engine or a search server). The electronic device can obtain search results corresponding to the search request and information about the search results (e.g., the access path of a web page including the search results) from the external device. The electronic device can determine at least a portion of the search results as search data.

[0077] In operation (310), the electronic device may extract a plurality of pieces of candidate data related to the query from among the source data based on obtaining the query. The candidate data may refer to at least a portion of the source data related to the query. As described later in operation (320), the plurality of candidate data may, for example, have the same or similar content.

[0078] In operation (320), the electronic device may select input data from among the extracted plurality of candidate data based on the contents of the plurality of candidate data. The input data may refer to data selected from among the plurality of candidate data to be used in generating a response to the query. The electronic device may select input data to be used in generating a response to the query based on the results of comparing the contents of the plurality of candidate data.

[0079] For example, if the contents of the first candidate data and the second candidate data are the same, the electronic device may select either the first candidate data or the second candidate data as input data. The selection of input data when the contents of the first candidate data and the second candidate data are the same will be described in more detail later in FIG. 5.

[0080] For example, if the contents of the first candidate data and the second candidate data are different, the electronic device may select either the first candidate data or the second candidate data, or both the first candidate data and the second candidate data, as input data. The selection of input data when the contents of the first candidate data and the second candidate data are different is described in more detail later in FIG. 6.

[0081] In operation (330), the electronic device can generate a response to the query by applying the query and selected input data to a generative model (e.g., the generative model (250) of FIG. 2).

[0082] The electronic device can convert the query and selected input data into an input format of the generative model. For example, the input data may include at least one of an image, text, video, or a document (e.g., a word processor, a slide show, or a spreadsheet). The electronic device can convert the input data into preprocessed input data having the input format of the generative model, as a preprocessing step for inputting the input data into the generative model.

[0083] A response to a query may include an answer to the question posed in the query. For example, a response to a query may include text based on natural language.

[0084] According to an embodiment, the electronic device may select one generative model from among a plurality of candidate generative models based on at least one of a query, source data, or input data. The electronic device may generate a response by applying the query and input data to the selected generative model.

[0085] For example, multiple candidate generative models can be selected based on the type of data included in the source data or input data among internal data, external device data, or search data. The electronic device can select a first generative model based on whether the source data or input data includes search data. The electronic device can select a second generative model based on whether the source data or input data does not include search data (e.g., whether the source data or input data includes at least one of internal data or external device data).

[0086] For example, multiple candidate generative models may individually correspond to multiple operations. The multiple operations may include at least one of a summary operation, an answer operation, a document generation operation, or a device control operation. For example, the operation may be selected based on an operation requested in a query. The electronic device may determine the operation requested in the query from among the multiple operations and select a generative model corresponding to the operation requested in the query.

[0087] In operation (340), the electronic device can determine a partial response derived from input data selected during the response.

[0088] An electronic device can obtain multiple candidate partial responses from a response. For example, the electronic device can classify components included in the response into multiple candidate partial responses based on grammatical and / or semantic criteria. For example, if the text included in the response includes multiple sentences, the electronic device can obtain each sentence as a single candidate partial response. For example, if the response includes an image, the electronic device can obtain each image as a single candidate partial response.

[0089] An electronic device may determine a similarity level between a plurality of candidate partial responses included in a response and input data. For example, the electronic device may determine the similarity between each candidate partial response and the input data. The electronic device may determine the similarity between each candidate partial response and the input data using a similarity calculation model. The similarity calculation model may refer to a model generated and / or trained to output an output (e.g., a similarity score) indicating a matching and / or similarity between the candidate partial response and the input data by being applied to an input including the candidate partial response and the input data. The similarity calculation model may be constructed based on a machine learning model (e.g., a neural network or LLM).

[0090] The electronic device may determine at least one candidate response as a partial response derived from the input data based on the similarity between each candidate partial response and the input data. For example, if the similarity between each candidate partial response and the input data is greater than or equal to a threshold similarity, the electronic device may determine that the candidate partial response is derived from the input data. If the similarity between each candidate partial response and the input data is less than the threshold similarity, the electronic device may determine that the candidate partial response is independent of the input data (e.g., not derived from the input data). If the similarity between the input data and all candidate partial responses is less than the threshold similarity, the electronic device may determine that none of the responses is derived from the input data. If the similarity between a specific candidate partial response and all input data is less than the threshold similarity, the electronic device may determine that the specific candidate partial response is not derived from the input data. If the specific candidate partial response is not derived from the input data, the electronic device may determine the specific candidate partial response as a partial response generated by a generative model, for example, as a partial response derived from data trained on the generative model.

[0091] According to an embodiment, when applying a plurality of input data to a generative model, the electronic device can determine a partial response derived from each input data among the responses. The electronic device can determine one partial response or two or more partial responses derived from each input data among the plurality of candidate partial responses.

[0092] In operation (350), the electronic device may display, through a display, a visual representation indicating information about at least one candidate data among a plurality of candidate data in an area corresponding to the determined partial response.

[0093] According to an embodiment, the electronic device may provide information about the input data in an area corresponding to a partial response derived from the input data.

[0094] The visual representation may indicate information about the origin of the input data. For example, the origin of the input data may include the hardware (e.g., an external device) or software (e.g., an application) used to acquire or process the input data, and / or the location (e.g., a directory) where the acquired input data is stored.

[0095] The visual representation may indicate whether the input data is internal data, external device data, or search data. For example, the visual representation may include a first visual representation indicating that the input data is internal data (e.g., a blue highlight), a second visual representation indicating that the input data is external device data (e.g., a yellow highlight), and a third visual representation indicating that the input data is search data (e.g., a red highlight). An example of a visual representation indicating that the input data is internal data or search data is described in more detail below in FIG. 4.

[0096] The visual representation may indicate at least one of an application used to obtain or process the input data, a directory (e.g., a folder) where the input data is stored, a path (e.g., a uniform resource locator (URL)) for accessing a page containing the input data, or an external device that shared the input data with the electronic device.

[0097] For example, if the input data is internal data, the visual representation may include an icon of an application used to obtain (e.g., receive or generate) and / or process (e.g., share, transmit, transform, or modify) the input data. If the input data is search data, the visual representation may include a connection path to a page (e.g., a web page) corresponding to the input data. If the input data is external device data, the visual representation may include a visual representation (e.g., an icon) indicating the external device that stores the input data and / or transmits the input data.

[0098] In various embodiments of the present disclosure, the visual representation is not limited to indicating information regarding the source of the input data. The visual representation may include a visual representation based on at least a portion of the input data. For example, the visual representation may include a preview of at least a portion of the input data. Examples of visual representations are described in more detail below in FIG. 14.

[0099] Although not explicitly shown in FIG. 3, the electronic device may further display a visual representation indicating information about candidate data that was not selected as input data among a plurality of candidate data, together with a visual representation indicating information about input data. For example, if the electronic device selects the first candidate data as input data among first candidate data and second candidate data having the same content and generates a response, the electronic device may display a second visual representation indicating information about the second candidate data, together with a first visual representation indicating information about the first candidate data. By displaying information about candidate data having the same content as the input data as well as the input data, the electronic device can provide the user with information about the input data as well as information about candidate data having the same content as the input data.

[0100] Although not explicitly shown in FIG. 3, if the electronic device determines that a portion of the response (e.g., a partial response) is not derived from the input data and / or is derived from data learned by the generative model, the electronic device may display a visual representation (e.g., an icon or highlight) in the area corresponding to the partial response indicating that the response is derived from data learned by the generative model. By displaying the partial response generated by the generative model among the responses, the electronic device may inform the user of the risk of hallucinations in which the generative model has errors and prompt a review of the partial response.

[0101] FIG. 4 illustrates an example of an interface for an electronic device to obtain a query and output a response according to various embodiments.

[0102] An electronic device according to an embodiment (e.g., electronic device (101) of FIG. 1) may provide an interface (400) for obtaining a query and outputting a response through a display and / or information about the interface (400). For example, the interface (400) may include a query input area (410), a source data designation area (420), a response output area (430), and / or an internal data designation area (440).

[0103] The query input area (410) may include an area for receiving user input for generating a query from a user.

[0104] The source data designation area (420) may include a first button (421) for designating at least a portion of internal data as source data, and a second button (422) for designating search data as source data.

[0105] The internal data designation area (440) may refer to an area for designating at least a portion of internal data as source data based on an application or directory. Through the internal data designation area (440), a user can designate at least a portion of the internal data to be used in generating a response to a query.

[0106] The response output area (430) may include an area for outputting a response to a query. As illustrated in FIG. 4, for example, the response may include a first candidate partial response (431), a second candidate partial response (432), a third candidate partial response (433), a fourth candidate partial response (434), a fifth candidate partial response (435), a sixth candidate partial response (436), a seventh candidate partial response (437), an eighth candidate partial response (438), and / or a ninth candidate partial response (439).

[0107] An electronic device can determine a correspondence between a plurality of candidate partial responses and input data. The correspondence between the plurality of candidate partial responses and the input data can indicate that a particular candidate partial response is derived from at least a portion of the input data.

[0108] The electronic device may determine the second candidate partial response (432) as a partial response derived from the input data, which is internal data. For example, the electronic device may display a visual representation indicating that the input data is internal data in the area corresponding to the second candidate partial response (432).

[0109] The electronic device may determine the fifth candidate partial response (435) and the eighth candidate partial response (438) as partial responses derived from the input data, which is search data. For example, the electronic device may display a visual representation indicating that the input data is search data in areas corresponding to the fifth candidate partial response (435) and the eighth candidate partial response (438).

[0110] The electronic device may determine the remaining candidate partial responses, for example, the first candidate partial response (431), the third candidate partial response (433), the fourth candidate partial response (434), the sixth candidate partial response (436), the seventh candidate partial response (437), and the ninth candidate partial response (439), as partial responses derived from the learning data of the generative model. As illustrated in FIG. 4 , the electronic device may limit the display of visual representations in areas corresponding to the remaining candidate partial responses. However, the present invention is not limited thereto, and although not explicitly illustrated in FIG. 4 , the electronic device may also display visual representations indicating that the remaining candidate partial responses are derived from the learning data of the generative model in areas corresponding to the remaining candidate partial responses.

[0111] FIG. 5 is a diagram illustrating an example of an operation of an electronic device according to various embodiments to select input data from among multiple candidate data having the same content.

[0112] According to one embodiment, an electronic device can store multiple pieces of data having the same content. For example, the electronic device can display a concert schedule image through an Internet application (521). The electronic device can store first data (511) including the concert schedule image obtained through the Internet application (521). The electronic device can store second data (512) including the concert schedule image through a photo application (522) based on obtaining a user input (e.g., inputting a screenshot) requesting storage of the concert schedule image. The electronic device can transmit third data (513) including the concert schedule image to another electronic device through a message application (523). As a result, multiple pieces of data having the same content as the concert schedule image (e.g., first data (511), second data (512), and third data (513)) can be stored.

[0113] An electronic device may obtain a query (540) (e.g., "When does the invited artist's performance start?"). The electronic device may extract a plurality of candidate data (510) related to the query (540) from source data, including first data (511), second data (512), and third data (513). Each candidate data may be mapped with information regarding an application used to obtain or process the candidate data. In FIG. 5 , the first data (511) may be mapped with information regarding an Internet application (521), the second data (512) may be mapped with information regarding a photo application (522), and the third data (513) may be mapped with information regarding a messaging application (523).

[0114] According to an embodiment, the electronic device may include an input data selection module (530) that selects input data (550) based on the contents of a plurality of candidate data (510). The input data selection module (530) may select input data (550) from among the plurality of candidate data (510).

[0115] For example, the input data selection module (530) may select one of the first candidate data or the second candidate data as input data (550) based on the fact that the contents of the first candidate data and the contents of the second candidate data are the same. The electronic device may select one candidate data among three or more candidate data as input data (550) based on the fact that the contents of the three or more candidate data are the same.

[0116] Whether the content of the first candidate data and the content of the second candidate data are the same can be determined based on the content queried in the query (540). For example, the query (540) (e.g., "When does the performance by the invited artist start?") can inquire about the start time of the performance by the invited artist. The first candidate data can include an image containing the content "The start time of the performance by the invited artist is 8:25." The second candidate data can include text containing the content "The start time of the performance by the invited artist is 8:25." Despite the difference in the format of the first candidate data and the format of the second candidate data, the electronic device can determine that the content queried in the query (540) included in the first candidate data and the content queried in the query (540) included in the second candidate data are the same. Even if there is a difference between the content queried in the query (540) among the contents included in the first candidate data and the content included in the second candidate data, the electronic device can determine that the first candidate data and the second candidate data have the same content with respect to the query (540) if the content queried in the query (540) included in the first candidate data and the content queried in the query (540) included in the second candidate data are the same.

[0117] Referring to FIG. 5, in an embodiment, the input data selection module (530) may select the first data (511) that is first acquired and / or processed by the electronic device among the first data (511), the second data (512), and the third data (513) as input data (550) based on the fact that the contents of the first data (511), the second data (512), and the third data (513) are the same.

[0118] According to an embodiment, the input data selection module (530) may map all information (e.g., information about a specific application) mapped to each candidate data to the selected input data (550) based on the fact that the contents of the plurality of candidate data (510) are the same. Referring to FIG. 5, the input data selection module (530) may map information about an Internet application (521), information about a photo application (522), and information about a message application (523) to the first data (511) selected as the input data (550).

[0119] The electronic device can generate a response (570) by applying a query (540) and input data (550) to a generative model (560). The electronic device can determine a partial response derived from the input data (550) among the responses (570). The electronic device can display a visual representation indicating information about the first candidate data and a visual representation indicating information about the second candidate data in an area corresponding to the partial response.

[0120] Referring to FIG. 5, the electronic device may determine that at least a portion of the generated response (570) is derived from the input data (550). The electronic device may display a visual representation indicating information about the input data (550) in an area corresponding to the response (570). If the electronic device selects the input data (550) based on the similarity of the contents of the plurality of candidate data (510), the electronic device may display a visual representation indicating information about each candidate data as information about the input data (550).

[0121] Referring to FIG. 5, the electronic device may display, in an area corresponding to a response (570), a first visual representation (581) regarding first data (511) (e.g., a visual representation indicating an Internet application (521)), a second visual representation (582) regarding second data (512) (e.g., a visual representation indicating a photo application (522)), and a third visual representation (583) regarding third data (513) (e.g., a visual representation indicating a message application (523)).

[0122] FIG. 6 is a diagram illustrating an example of an operation of an electronic device according to various embodiments to select input data from among a plurality of candidate data having different contents.

[0123] In one embodiment, the electronic device may store multiple pieces of data having different content. For example, the electronic device may store announcements regarding an annual swimming competition.

[0124] An electronic device can obtain a query (640) (e.g., "Tell me the entry fee for a swimming competition and when it is"). For example, the source data may include data stored in a first device (601) (e.g., external device data) and data stored in a second device (602) (e.g., internal data). For example, the first device (601) may be an external device of the electronic device, and the second device (602) may be an electronic device. The electronic device may extract first data (611), second data (612), and third data (613) from among the data stored in the first device (601), and fourth data (614) and fifth data (615) from among the data stored in the second device (602), as a plurality of candidate data (610) related to the query (640) from the source data.

[0125] In FIG. 6, first data (611) may be mapped with information about a first device (601) and / or a first directory (621) in which the first data (611) is stored, second data (612) may be mapped with information about a second directory (622) in which the first device (601) and / or the second data (612) are stored, and third data (613) may be mapped with information about a third directory (623) in which the first device (601) and / or the second data (612) are stored. Fourth data (614) may be mapped with information about a second device (602) and / or a fourth directory (624) in which the fourth data (614) is stored, and fifth data (615) may be mapped with information about a second device (602) and / or a fifth directory (625) in which the fifth data (615) is stored.

[0126] According to an embodiment, the electronic device may include an input data selection module (630) that selects input data (650) based on the contents of a plurality of candidate data (610). The input data selection module (630) may select input data (650) from among the plurality of candidate data (610).

[0127] For example, the input data selection module (630) may select candidate data that has been acquired (e.g., created, received, or stored) or processed (e.g., shared, transmitted, or modified) more recently than other candidate data among the first candidate data and the second candidate data, as input data (650), based on the difference between the contents of the first candidate data and the contents of the second candidate data.

[0128] For example, as described above in FIG. 5, whether the content of the first candidate data and the content of the second candidate data are the same or different can be determined based on the content queried in the query (640). For example, the query (640) (e.g., "Tell me the entry fee for the swimming competition and the date") can inquire about the entry fee for the swimming competition and the date of the swimming competition. The first candidate data can have the content of 'The entry fee for the swimming competition is 20,000 won and the date of the swimming competition is June 29th.' The second candidate data can have the content of 'The entry fee for the swimming competition is 30,000 won and the date of the swimming competition is June 17th.' The electronic device can determine that the content queried in the query (640) included in the first candidate data and the content queried in the query (640) included in the second candidate data are different from each other.

[0129] For example, the input data selection module (630) may include an acquisition timepoint and / or a processing timepoint of each candidate data. The acquisition timepoint of each candidate data may include a timepoint when the corresponding candidate data is generated, or a timepoint when the corresponding candidate data is stored in a device (e.g., the first device (601) or the second device (602)). The processing timepoint of each candidate data may include a timepoint when the corresponding candidate data is shared or transmitted to another electronic device, or a timepoint when the corresponding candidate data is modified. If there are multiple processing timepoints of the corresponding candidate data according to multiple processing operations for each candidate data, the input data (650) may be selected based on the last processing timepoint among the multiple processing timepoints.

[0130] The electronic device may select the most recently generated or processed candidate data among three or more candidate data as input data (650) based on the differences in the contents of the three or more candidate data. Referring to FIG. 6, the input data selection module (630) may select the most recently generated third data (613) among the first data (611), the second data (612), the third data (613), the fourth data (614), and the fifth data (615) as input data (650).

[0131] In FIG. 6, the electronic device mainly describes selecting input data from among a plurality of candidate data having different contents based on the acquisition time and / or processing time, but is not limited thereto. For example, the electronic device may select input data based on the quality and / or confidence of the plurality of candidate data. When selecting input data from among a plurality of images, the electronic device may select the input data from among the plurality of images based on at least one of the image quality (resolution), shaking, and / or objects appearing in the image.

[0132] The input data selection module (630) can map information about the selected input data (650) to the input data (650) based on the differences in the contents of the plurality of candidate data (610). Unlike the case where the contents of the plurality of candidate data (610) are the same, where information about other candidate data is mapped to the input data (650), the input data selection module (630) can map only information about the input data (650) to the input data (650) when the contents of the plurality of candidate data (610) are different. Referring to FIG. 6, the input data selection module (630) can map information about the first device (601) and / or information about the third directory (623) to the third data (613) selected as the input data (650).

[0133] The electronic device can generate a response (670) by applying a query (640) and input data (650) to a generative model (660). The electronic device can determine a partial response derived from the input data (650) among the responses (670). The electronic device can display a visual representation indicating information about the input data (650) in an area corresponding to the partial response.

[0134] Referring to FIG. 6, the electronic device may determine that at least a portion of the generated response (670) is derived from the input data (650). The electronic device may display a visual representation indicating information about the input data (650) in an area corresponding to the response (670). The electronic device may display a visual representation (681) indicating the first device (601) and / or a visual representation (682) indicating the third directory (623) based on information mapped to the input data (650) (e.g., information about the first device (601) and / or information about the third directory (623)).

[0135] In Fig. 6, the electronic device mainly describes, but is not limited to, selecting the most recently generated or processed candidate data among the plurality of candidate data as input data when the contents of the plurality of candidate data are different from each other.

[0136] Although not explicitly shown in FIG. 6, the electronic device may determine whether to select some of the plurality of candidate data as input data or all of the plurality of candidate data as input data based on the category(es) of the contents of the plurality of candidate data when the contents of the plurality of candidate data are different from each other.

[0137] According to an embodiment, the electronic device may select the most recently generated or processed candidate data among the first candidate data and the second candidate data as input data, based on the content of the first candidate data and the content of the second candidate data being in the same category as described above.

[0138] According to an embodiment, the electronic device may select the first candidate data and the second candidate data as input data based on the content of the first candidate data and the content of the second candidate data being different categories.

[0139] In an embodiment, the category of the content of the candidate data may correspond to one of multiple items queried in the query. For example, multiple items may be queried in the query. A query (e.g., "Tell me the entry fee for the swimming competition and the date") may query a first item (e.g., the entry fee for the swimming competition) and a second item (e.g., the date of the swimming competition). The electronic device may determine the category of the content of each of the first candidate data and the second candidate data as either the first item or the second item. The first candidate data may have the content "The entry fee for the swimming competition is 30,000 won." The second candidate data may have the content "The date of the swimming competition is June 17th." If the electronic device determines the category of the first candidate data as the first item and the category of the second candidate data as the second item, both the first candidate data and the second candidate data may be selected as input data.

[0140] According to an embodiment, the electronic device may determine that the category of the content of the first candidate data and the category of the content of the second candidate data are different if the content of the first candidate data and the content of the second candidate data are compatible with the query. The fact that the content of the first candidate data and the content of the second candidate data are compatible with the query may mean that even if both the portion of the content of the first candidate data corresponding to the query and the portion of the content of the second candidate data corresponding to the query are included, they do not contradict each other.

[0141] For example, a query (e.g., "Create a Jeju Island experiential learning report") may request the creation of a Jeju Island experiential learning report. The first candidate data may include "an image of the sea of ​​Jeju Island." The second candidate data may include "an image of Jeju International Airport." When creating a Jeju Island experiential learning report, the electronic device may determine that the categories of the contents of the first candidate data and the second candidate data are different based on the compatibility of the "image of the sea of ​​Jeju Island" of the first candidate data and the "image of Jeju International Airport" of the second candidate data, and may select both the first candidate data and the second candidate data as input data.

[0142] According to an embodiment, an electronic device may generate a response by applying a generative model to input data and a query including first candidate data and second candidate data. The electronic device may determine a first partial response derived from the first candidate data and a second partial response derived from the second candidate data among the responses. The electronic device may display a first visual representation indicating information about the first candidate data in an area corresponding to the first partial response. The electronic device may display a second visual representation indicating information about the second candidate data in an area corresponding to the second partial response.

[0143] FIG. 7A illustrates an example of a response to a query generated by an electronic device using specified source data according to various embodiments. FIG. 7B illustrates an example of a response to a query generated by an electronic device when source data is not specified according to various embodiments.

[0144] An electronic device according to an embodiment (e.g., electronic device (101) of FIG. 1) can obtain a query (e.g., "Tell me the entry fee for the swimming competition and how many days it is") (710a, 710b).

[0145] Referring to FIG. 7A, in an embodiment, an electronic device may designate source data as at least a portion of internal data or external device data. The electronic device may determine, among the source data, a 2024 swimming competition notice file containing the content of the 2024 swimming competition notice as input data, and generate a response (720a) (e.g., "The participation fee is 30,000 won, and the date is June 17th") based on the input data. The electronic device may output the response (720a) (e.g., play a sound corresponding to the response (720a), or display a visual representation corresponding to the response (720a). The electronic device may also display, in an area corresponding to the response (720a), a visual representation indicating information about the input data (e.g., the 2024 swimming competition notice file). In FIG. 7a, a visual representation indicating information about input data may include, for example, a first visual representation (731a) indicating an external device where the input data is stored and a second visual representation (732a) including at least a portion of the input data (e.g., a preview of the input data).

[0146] Referring to FIG. 7b, the electronic device may not designate internal data or external device data as source data (e.g., only search data may be designated as source data), or may not designate source data. The electronic device may determine that additional information (e.g., information for specifying a swimming competition) is required to specify the content queried in the query (710b) (e.g., the entry fee for a swimming competition or the date of the swimming competition). The electronic device may generate a response (720b) ​​requesting the additional information.

[0147] FIG. 8 is a diagram illustrating an example of an operation of an electronic device according to various embodiments to display a visual representation indicating information about a generative model used to generate a response.

[0148] An electronic device according to an embodiment (e.g., electronic device (101) of FIG. 1) may obtain a query (810) (e.g., “Why is there a lot of basalt in Jeju Island?”). The electronic device may designate search data as source data based on analysis of the query (810) and / or user input. Based on whether the query (810) inquires about public information (e.g., information that is not private information) and / or whether the search data is designated as source data, the electronic device may select a second generative model from among a first generative model that processes private information and a second generative model that processes public information. The electronic device may generate a response (820) using the second generative model. The electronic device may display a visual representation (830) (e.g., an icon) indicating the second generative model in an area corresponding to the response (820).

[0149] Although not explicitly shown in FIG. 8, if the electronic device generates a response (820) using the first generative model, it may display a visual representation (840) indicating the first generative model instead of a visual representation (830) indicating the second generative model.

[0150] FIG. 9 illustrates an example of an interface for obtaining a query and outputting a response when an electronic device designates user-provided data and search data as source data according to various embodiments.

[0151] An electronic device according to an embodiment (e.g., electronic device (101) of FIG. 1) may provide an interface (900) (e.g., interface (400) of FIG. 4) for obtaining user input regarding obtaining a query and / or generating a response, and outputting the generated response, and / or information about the interface (900).

[0152] According to an embodiment, the interface (900) may include a query input area (910), a source data specification area (920), a response output area (930), a user-provided data specification area (940), and / or a template provision area (950).

[0153] According to an embodiment, the query input area (910) may include an area for receiving user input for generating a query from a user.

[0154] According to an embodiment, the source data designation area (920) may include a button for designating at least a portion of user-provided data (e.g., internal data or external device data) as source data, and a button for designating search data as source data.

[0155] According to an embodiment, the response output area (930) may include an area for outputting a response to a query. Examples of responses shown through the response output area (930) are described in more detail below.

[0156] According to an embodiment, the user-provided data designation area (940) may include a directory designation area (942) and an application and device designation area (941).

[0157] According to an embodiment, the directory designation area (942) may refer to an area for designating source data based on a directory (e.g., a folder) and / or a file. For example, if a specific directory is added to the directory designation area (942), the electronic device may designate data (e.g., a file) stored in the specific directory as source data. If a specific file is added to the directory designation area (942), the electronic device may designate the specific file as source data.

[0158] According to an embodiment, the application and device designation area (941) may refer to an area for designating source data based on an application or a device (e.g., an external device). In the application and device designation area (941), a list of application(s) and / or a list of device(s) may be displayed, and data related to a specific application and / or a specific device may be designated as source data based on a user input for selecting a specific application and / or a specific device. For example, in the application and device designation area (941), the electronic device may designate data acquired or processed through a specific application among internal data as source data based on a user input for enabling data of a specific application. For example, in the application and device designation area (941), the electronic device may designate external device data corresponding to a specific device as source data based on a user input for enabling data of a specific device. Conversely, the electronic device may exclude (e.g., remove) data of a specific application and / or a specific device from the source data based on a user input for disabling data of a specific application and / or a specific device. The application and device designation area (941) is described in more detail later in FIG. 11.

[0159] According to an embodiment, the template providing area (950) may include a recommended template area (951) and a style area (952).

[0160] According to an embodiment, the recommendation template area (951) may refer to an area for providing templates recommended based on a query and / or source data among templates. When a query requests a document creation operation, the electronic device may select one or more recommendation templates from among a plurality of candidate templates based on the query and / or source data. For example, the electronic device may display a list of one or more selected recommendation templates in the recommendation template area (951).

[0161] According to an embodiment, an electronic device may select a recommended template independently of user information or with greater consideration of user information than when user-provided data is not designated as source data, when user-provided data is designated as source data. The user information may include information indicating the user's preference for a template (e.g., a candidate template). For example, when user-provided data is designated as source data, the electronic device may select a recommended template considering the user's preference, and when user-provided data is not designated as source data, the electronic device may select a recommended template without considering the user's preference.

[0162] According to an embodiment, the style area (952) may refer to an area for providing the format of a response. The format of the response may include the format of text included in the response (e.g., font or size) and effects applied to images included in the response (e.g., filters).

[0163] In FIG. 9, the electronic device can designate at least a portion of user-provided data and search data as source data based on the source data designation area (920). Among the user-provided data, the electronic device can designate directories and / or files added to the directory designation area (942) as source data. The electronic device can designate data corresponding to enabled applications and / or external devices as source data through the application and device designation area (941). Thereafter, the electronic device can obtain a query (e.g., "Create an experiential learning report") through the query input area (910). The electronic device can generate a response to the query (e.g., a document including an experiential learning report) based on the source data and the query.

[0164] Referring to FIG. 9, a response may be generated as a document including multiple images and multiple texts. For example, the response may include a first image (961), a second image (962), a third image (963), and a fourth image (964), and may include a first text (965) and a second text (966). Each image (e.g., the first image (961), the second image (962), the third image (963), or the fourth image (964)) or each text (e.g., the first text (965) or the second text (966)) included in the response may be included in the response as a partial response.

[0165] For example, the electronic device may determine each of the first image (961), the second image (962), the third image (963), and the fourth image (964) as partial responses derived from input data obtained from a photo application. The electronic device may display a visual representation (971) indicating the photo application in an area corresponding to each of the first image (961), the second image (962), the third image (963), and the fourth image (964).

[0166] For example, the electronic device may determine the first text (965) as a partial response derived from the search data. The electronic device may display a visual representation (972) indicating the search data and / or a web page corresponding to the search data in an area corresponding to the first text (965).

[0167] For example, the electronic device may determine the second text (966) as a partial response generated by the generative model (e.g., a partial response derived from the learned data of the generative model, a partial response not derived from the input data). The electronic device may display a visual representation (973) indicating that the second text (966) was generated by the generative model in an area corresponding to the second text (966).

[0168] Although not explicitly illustrated in FIG. 9, the electronic device may generate multiple versions of responses to a query (e.g., "Create an experiential learning report"). For example, the electronic device may generate a first response of a first version and a second response of a second version. The first version, also referred to as a private version, may indicate that the first response uses a greater proportion of user-provided data than the second response. The second version, also referred to as a public version, may indicate that the second response uses a lesser proportion of user-provided data than the first response.

[0169] FIG. 10 illustrates an example of an interface for obtaining a query and outputting a response when an electronic device, according to various embodiments, specifies search data as source data rather than user-provided data as source data.

[0170] An electronic device according to an embodiment (e.g., electronic device (101) of FIG. 1) may provide an interface (1000) (e.g., interface (400) of FIG. 4 or interface (900) of FIG. 9) for obtaining user input regarding obtaining a query and / or generating a response, and outputting the generated response, and / or information about the interface (1000).

[0171] According to an embodiment, the interface (1000) may include a query input area (1010), a source data specification area (1020), a response output area (1030), an internal data specification area (1040), and a template provision area (1050). Each area may be the same as or similar to that described in FIG. 4 and / or FIG. 9.

[0172] In FIG. 10, the electronic device may designate search data as source data instead of user-provided data as source data based on the source data designation area (1020). Thereafter, the electronic device may obtain a query (e.g., "Create an experiential learning report") through the query input area (1010). Based on the source data and the query, the electronic device may generate a response to the query (e.g., a document including the experiential learning report).

[0173] Referring to FIG. 10, a response may be generated as a document including multiple images and multiple texts. For example, the response may include a first image (1061) and a second image (1062), and may include a first text (1063) and a second text (1064). Each image (e.g., the first image (1061) or the second image (1062)) or each text (e.g., the first text (1063) or the second text (1064)) included in the response may be included in the response as a partial response.

[0174] For example, the electronic device may determine each of the first image (1061) and the second image (1062) as a partial response derived from the search data. The electronic device may display a visual representation (1071) indicating the search data and / or a web page in an area corresponding to each of the first image (1061) and the second image (1062).

[0175] For example, the electronic device may determine the first text (1063) as a partial response derived from the search data. The electronic device may display a visual representation (1071) indicating the search data and / or a web page in an area corresponding to the first text (1063).

[0176] For example, the electronic device may determine that the second text (1064) is a partial response generated by the generative model (e.g., a partial response derived from the learned data of the generative model, or a partial response not derived from the input data). The electronic device may display a visual representation (1073) indicating that the second text (1064) was generated by the generative model in an area corresponding to the second text (1064).

[0177] FIG. 11 illustrates an example of a screen for specifying data related to a specific application as source data according to various embodiments.

[0178] An electronic device according to an embodiment (e.g., the electronic device (101) of FIG. 1) may obtain user input for setting whether to use data related to a given application as source data on a screen (1100) for setting permissions for each application. The data related to the application may include data obtained or processed through the application.

[0179] Referring to FIG. 11, an electronic device may display a screen (1100) for setting permissions related to a specific application (e.g., a photo application). For example, the screen (1100) may include an area (1110) for setting the use of data related to a specific application as input data and / or source data for a generative model.

[0180] According to an embodiment, in the region (1110), the electronic device may designate data related to a specific application as source data if internal data (or user-provided data) is designated as the source data based on a user input that makes data related to a specific application available as input data of a generative model and / or as source data. As another example, the electronic device may exclude data related to a specific application from the source data even if internal data (or user-provided data) is designated as the source data based on a user input that makes data related to a specific application unavailable as input data of a generative model and / or as source data.

[0181] FIG. 12 illustrates an example of a screen for specifying source data based on a directory in an electronic device according to various embodiments.

[0182] An electronic device according to an embodiment (e.g., electronic device (101) of FIG. 1) may obtain user input for setting whether to use a specific directory and / or a specific file as source data.

[0183] By way of example, the electronic device may, in response to a user input selecting a file application in the application and device designation area of ​​the interface (e.g., the application and device designation area (941) of FIG. 9), display a screen (1200) for selecting at least some of the internal data of the electronic device based on directories and / or files.

[0184] Referring to FIG. 12, the screen (1200) may include a directory list of at least some of the internal data of the electronic device. For example, the screen (1200) may include a plurality of buttons individually corresponding to a plurality of directories. The electronic device may designate data stored in a specific directory as source data in response to a user input of selecting a button corresponding to a specific directory (e.g., directory A). In FIG. 12, directories A, B, and C are designated as source data, and the remaining directories (e.g., directory D, directory F, directory G, directory H, directory I, directory J, directory K, directory L, directory M, directory O, or directory P) are illustrated as not being designated as source data.

[0185] FIG. 13 is a diagram illustrating examples of screens that include responses of electronic devices according to various embodiments.

[0186] An electronic device according to an embodiment (e.g., electronic device (101) of FIG. 1) may display a response (1320) (e.g., "It starts at 8:25") generated based on a query (1310) (e.g., "When does the invited artist performance start?").

[0187] On the screen (1301), the electronic device can display a visual representation (1330) in an area corresponding to the generated response (1320) that indicates information about the input data from which the response (1320) originated (e.g., a photo application).

[0188] On screen (1302), the electronic device may display at least a portion (1350) of input data based on obtaining user input (1340) for the displayed visual representation (1330). Referring to FIG. 13, the electronic device may display a preview including at least a portion (1350) of the input data in an area corresponding to the response (1320).

[0189] However, and not limited thereto, the electronic device may output (e.g., display) input data by executing an application (e.g., a photo application) that provides input data in response (1320) to user input (1340).

[0190] An electronic device according to an embodiment may provide a user with easy verification of a response (1320) or partial response (1320) based on input data by providing at least a portion of the input data through a user input (1340) to a visual representation (1330).

[0191] Although not explicitly shown in FIG. 13, the electronic device may display one of the screens (1301) or (1302) based on information about the display of the electronic device. The information about the display may include, for example, the resolution of the display, the size of the display, or the resolution and / or size of a window allocated to display the interface (or response (1320)).

[0192] For example, if the electronic device is a smartphone having a display having a resolution below a threshold resolution (e.g., a small display), the electronic device may display a screen (1301) to display a response (1320). On the screen (1301), a visual representation (1330) may be displayed as a visual representation indicating information about the input data. If the electronic device is a laptop having a display having a resolution above a threshold resolution (e.g., a large display), the electronic device may display a screen (1302) when displaying a response (1320). On the screen (1302), the visual representation (1330) and at least a portion (1350) of the input data may be displayed as a visual representation indicating information about the input data.

[0193] FIG. 14 is a diagram illustrating an example configuration for performing operations of an electronic device according to various embodiments.

[0194] An electronic device according to an embodiment (e.g., electronic device (101) of FIG. 1) may include a query analysis module (1410), an input data selection module (1420) (e.g., input data selection module (530) of FIG. 5 or input data selection module (630) of FIG. 6), a generative model interface module (1430), a response generation module (1440), and an input data-partial response correspondence module (1450). Each of these modules may include various circuits (e.g., processing circuits) and / or executable program instructions.

[0195] According to an embodiment, the query analysis module (1410) may acquire a query (1401) and / or analyze the acquired query (1401). As described above, the result of analyzing the acquired query (1401) may be used to specify source data, extract candidate data, select input data, and / or select a generative model.

[0196] According to an embodiment, the input data selection module (1420) may obtain the result of analyzing the query (1401) from the query analysis module (1410) and select input data from among the specified source data and / or a plurality of extracted candidate data. For example, the source data may be specified as at least a portion of internal data (1421), at least a portion of external device data (1422), and / or search data (1423). The input data selection module (1420) may transmit the selected input data to the generative model interface module (1430).

[0197] According to an embodiment, the generative model interface module (1430) may receive a query (1401) and / or a result of analyzing the query (1401) from the query analysis module (1410). The generative model interface module (1430) may obtain input data from the input data selection module (1420). For example, the generative model interface module (1430) may select a generative model from among a plurality of candidate generative models and / or obtain information about the selected generative model. The generative model interface module (1430) may transmit information about the query (1401), the input data, and / or the generative model to the response generation module (1440).

[0198] According to an embodiment, the response generation module (1440) can generate a response (1441) by applying a query (1401) and input data to a generative model based on information received from the generative model interface module (1430). The response generation module (1440) can transmit the generated response (1441) to the input data-partial response correspondence module (1450).

[0199] According to an embodiment, the input data-partial response correspondence module (1450) may receive input data from the response generation module (1440) and / or the input data selection module (1420). The input data-partial response correspondence module (1450) may receive a response (1441) from the response generation module (1440). The input data-partial response correspondence module (1450) may determine a correspondence relationship (1451) between the input data (or a plurality of partial input data) and the plurality of partial responses. As described above with reference to FIGS. 2 to 13 , a visual representation indicating information about the input data may be displayed in a peripheral area of ​​an area where a partial response corresponding to the input data (e.g., a partial response derived from the input data) is displayed.

[0200] According to an embodiment, the electronic device (101) comprises: a display (160); at least one processor (120) including a processing circuit; And a memory (130) for storing instructions, wherein the at least one processor (120) is configured to individually and / or collectively execute the instructions, and causes the electronic device (101) to extract a plurality of pieces of candidate data (510; 610) related to the query (540; 640; 710a; 710b; 810; 1310; 1401) from among the source data based on obtaining a query (540; 640; 710a; 710b; 810; 1310; 1401) for querying source data, and select input data (550; 650) from among the extracted plurality of candidate data (510; 610) based on contents of the plurality of candidate data, and to execute the query (540; 640; 710a; 710b; 810; 1310; 1401). By applying the selected input data (550; 650) to the generative model (250; 560; 660) and the query (540; 640; 710a; 710b; 810; 1310; 1401), a response (570; 670; 720a; 720b; 820; 1320; 1441) to the query (540; 640; 710a; 710b; 810; 1310; 1401), and determining a partial response derived from the selected input data (550; 650) among the responses (570; 670; 720a; 720b; 820; 1320; 1441), and displaying the determined partial response through the display (160). A visual representation (581; 582; 583; 681; 682; 1330) indicating information about the selected input data (550; 650) in the region can be displayed.

[0201] The visual representation (581; 582; 583; 681; 682; 1330) shown above may indicate at least one of an application used to obtain or process the input data (550; 650), a directory where the input data (550; 650) is stored, a connection path for accessing a page containing the input data (550; 650), or an external device that shared the input data (550; 650) with the electronic device (101).

[0202] The at least one processor (120) may be configured to individually and / or collectively cause the electronic device (101) to select one of the first candidate data or the second candidate data as the input data (550; 650) based on the content of the first candidate data being the same as the content of the second candidate data, and to display a visual representation indicating information about the first candidate data and a visual representation indicating information about the second candidate data in an area corresponding to the partial response.

[0203] The at least one processor (120) may be configured, individually and / or collectively, to cause the electronic device (101) to select, as input data (550; 650), candidate data that is acquired or processed more recently than other candidate data among the first candidate data and the second candidate data, based on differences between the contents of the first candidate data and the contents of the second candidate data.

[0204] The at least one processor (120) may be configured to, individually and / or collectively, cause the electronic device (101) to select the first candidate data and the second candidate data as input data (550; 650), determine a first partial response derived from the first candidate data and a second partial response derived from the second candidate data among the responses (570; 670; 720a; 720b; 820; 1320; 1441), and display a first visual representation indicating information about the first candidate data in an area corresponding to the first partial response and a second visual representation indicating information about the second candidate data in an area corresponding to the second partial response.

[0205] The at least one processor (120) may be configured, individually and / or collectively, to cause the electronic device (101) to display at least a portion of the input data (550; 650) based on obtaining input for the displayed visual representation (581; 582; 583; 681; 682; 1330).

[0206] The at least one processor (120) may be configured to individually and / or collectively cause the electronic device (101) to determine a similarity level between a plurality of candidate partial responses included in the response (570; 670; 720a; 720b; 820; 1320; 1441) and the input data (550; 650), and to determine at least one candidate response (570; 670; 720a; 720b; 820; 1320; 1441) as the partial response derived from the input data (550; 650) based on the similarity level between each candidate partial response and the input data (550; 650).

[0207] The at least one processor (120) may be configured to, individually and / or collectively, cause the electronic device (101) to determine, based on an input, the source data as at least one of at least a portion of internal data stored in the memory (130), at least a portion of external device data stored in another electronic device connected to the electronic device (101), or search data obtainable through a search server.

[0208] The at least one processor (120) may be configured to, individually and / or collectively, cause the electronic device (101) to determine the source data as internal data stored in the memory (130) of the electronic device (101) based on the query (540; 640; 710a; 710b; 810; 1310; 1401) inquiring about private information of the user.

[0209] The at least one processor (120) individually and / or collectively causes the electronic device (101) to obtain summary data of internal data stored in the memory (130), and, based on obtaining the query (540; 640; 710a; 710b; 810; 1310; 1401), determine whether the query (540; 640; 710a; 710b; 810; 1310; 1401) is answerable by the internal data using the summary data and the query (540; 640; 710a; 710b; 810; 1310; 1401), and determine whether the query (540; 640; 710a; 710b; 810; 1310; 1401) is answerable by the internal data. Based on what is answerable, the source data including the internal data can be specified, and based on what is not answerable by the internal data, the query (540; 640; 710a; 710b; 810; 1310; 1401) can be configured to specify the source data including at least one of external device data or search data.

[0210] A method performed by an electronic device (101) according to an exemplary embodiment may include: an operation (310) of extracting a plurality of pieces of candidate data related to a query from among source data based on obtaining a query for querying source data; an operation (320) of selecting input data from among the plurality of extracted candidate data based on contents of the plurality of candidate data; an operation (330) of generating a response to the query by applying the query and the selected input data to a generative model; an operation (340) of determining a partial response derived from the selected input data from among the responses; and an operation (350) of displaying a visual representation indicating information about the selected input data in an area corresponding to the determined partial response through a display.

[0211] The visual representation shown above indicates at least one of an application used to obtain or process the input data, a directory where the input data is stored, a connection path for accessing a page containing the input data, or an external device that shared the input data with the electronic device.

[0212] The operation (320) of selecting the input data may include an operation of selecting one of the first candidate data or the second candidate data as the input data based on the content of the first candidate data being the same as the content of the second candidate data, and the operation (350) of displaying the visual representation may include an operation of displaying a visual representation indicating information about the first candidate data and a visual representation indicating information about the second candidate data in an area corresponding to the partial response.

[0213] The operation (320) of selecting the input data may include an operation of selecting candidate data that is acquired or processed more recently than other candidate data among the first candidate data and the second candidate data as input data, based on the difference between the contents of the first candidate data and the contents of the second candidate data.

[0214] The operation (320) of selecting the input data may include an operation of selecting the first candidate data and the second candidate data as input data based on the fact that the contents of the first candidate data and the contents of the second candidate data are different categories, the operation (340) of determining the partial response may include an operation of determining a first partial response derived from the first candidate data and a second partial response derived from the second candidate data among the responses, and the operation (350) of displaying the visual representation may include an operation of displaying a first visual representation indicating information about the first candidate data in an area corresponding to the first partial response and a second visual representation indicating information about the second candidate data in an area corresponding to the second partial response.

[0215] The method may further include an action of displaying at least a portion of the input data based on obtaining input for the displayed visual representation.

[0216] The operation (340) of determining the partial response may include an operation of determining a similarity level between a plurality of candidate partial responses included in the response and the input data; and an operation of determining at least one candidate response as the partial response derived from the input data based on the similarity level between each candidate partial response and the input data.

[0217] The method may further include an operation of determining, based on the input, the source data as at least one of at least a portion of internal data stored in a memory of the electronic device, at least a portion of external device data stored in another electronic device connected to the electronic device, or search data obtainable through a search server.

[0218] The method may further include an action of determining the source data as internal data stored in a memory of the electronic device based on the query querying the user's private information.

[0219] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

[0220] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another component (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0221] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0222] Various embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0223] According to one embodiment, the method according to various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play Store™) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0224] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0225] The embodiments described above may be implemented using hardware components, software components, and / or a combination of hardware components and software components. For example, the devices, methods, and components described in the embodiments may be implemented using a general-purpose computer or a special-purpose computer, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding to them. The processing device may execute an operating system (OS) and software applications running on the operating system. Furthermore, the processing device may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device is sometimes described as being used alone; however, one of ordinary skill in the art will recognize that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, a processing unit may include multiple processors, or a processor and a controller. Other processing configurations, such as parallel processors, are also possible.

[0226] Software may include computer programs, codes, instructions, or a combination of one or more of these, which may configure a processing device to perform a desired operation or may independently or collectively command the processing device. The software and / or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, or computer storage medium or device for interpretation by the processing device or for providing instructions or data to the processing device. The software may also be distributed over networked computer systems and stored or executed in a distributed manner. The software and data may be stored on a computer-readable recording medium.

[0227] The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program commands, data files, data structures, etc., alone or in combination, and the program commands recorded on the medium may be those specially designed and configured for the embodiment or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program commands such as ROMs, RAMs, and flash memories. Examples of program commands include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc.

[0228] The hardware device described above may be configured to operate as one or more software modules to perform the operations of various embodiments, and vice versa.

[0229] While the present disclosure has been illustrated and described with reference to various embodiments, it will be understood that the various embodiments are illustrative and not restrictive. It will be further understood that various changes in form and detail may be made by those skilled in the art without departing from the spirit and scope of the present disclosure, including the appended claims and their equivalents. Furthermore, it will be understood that any of the embodiment(s) described herein can be utilized in conjunction with other embodiment(s) described herein.

Claims

1. In an electronic device (101), display (160); At least one processor (120) comprising a processing circuit; and Includes a memory (130) for storing instructions, At least one processor (120) above, are individually and / or collectively configured to execute the above instructions, The above electronic device (101) causes, Based on obtaining a query (540; 640; 710a; 710b; 810; 1310; 1401) that queries source data, a plurality of pieces of candidate data (510; 610) related to the query (540; 640; 710a; 710b; 810; 1310; 1401) are extracted from the source data, Among the plurality of candidate data (510; 610) extracted above, input data (550; 650) is selected based on the contents of the plurality of candidate data, By applying the above query (540; 640; 710a; 710b; 810; 1310; 1401) and the above selected input data (550; 650) to a generative model (250; 560; 660), a response (570; 670; 720a; 720b; 820; 1320; 1441) to the above query (540; 640; 710a; 710b; 810; 1310; 1401) is generated, Determine the partial response derived from the selected input data (550; 650) among the above responses (570; 670; 720a; 720b; 820; 1320; 1441), Through the above display (160), a visual representation (581; 582; 583; 681; 682; 1330) indicating information about the selected input data (550; 650) in the area corresponding to the determined partial response is displayed. To do, Electronic device (101).

2. In paragraph 1, The visual representations shown above (581; 582; 583; 681; 682; 1330) are: Indicates at least one of an application used to obtain or process the input data (550; 650), a directory where the input data (550; 650) is stored, a connection path for accessing a page containing the input data (550; 650), or an external device that shared the input data (550; 650) with the electronic device (101). Electronic device (101).

3. In any one of paragraphs 1 and 2, The at least one processor (120) individually and / or collectively causes the electronic device (101) to: Based on the fact that the contents of the first candidate data and the contents of the second candidate data are the same, one of the first candidate data or the second candidate data is selected as the input data (550; 650), configured to display a visual representation indicating information about the first candidate data and a visual representation indicating information about the second candidate data in an area corresponding to the partial response. Electronic device (101).

4. In any one of paragraphs 1 to 3, The at least one processor (120) individually and / or collectively causes the electronic device (101) to: Based on the difference between the contents of the first candidate data and the contents of the second candidate data, the candidate data that is acquired or processed more recently than other candidate data among the first candidate data and the second candidate data is selected as input data (550; 650). Electronic device (101).

5. In any one of paragraphs 1 to 4, The at least one processor (120) individually and / or collectively causes the electronic device (101) to: Based on the fact that the contents of the first candidate data and the contents of the second candidate data are different categories, the first candidate data and the second candidate data are selected as input data (550; 650). Among the above responses (570; 670; 720a; 720b; 820; 1320; 1441), a first partial response derived from the first candidate data and a second partial response derived from the second candidate data are determined, A first visual representation indicating information about the first candidate data in an area corresponding to the first partial response and a second visual representation indicating information about the second candidate data in an area corresponding to the second partial response, configured to display the first visual representation indicating information about the first candidate data in an area corresponding to the second partial response. Electronic device (101).

6. In any one of paragraphs 1 to 5, The at least one processor (120) individually and / or collectively causes the electronic device (101) to: Based on obtaining input for the visual representation (581; 582; 583; 681; 682; 1330) indicated above, configured to display at least a portion of the input data (550; 650), Electronic device (101).

7. In any one of paragraphs 1 to 6, The at least one processor (120) individually and / or collectively causes the electronic device (101) to: Determine the similarity level between the multiple candidate partial responses included in the above responses (570; 670; 720a; 720b; 820; 1320; 1441) and the input data (550; 650), Determine at least one candidate response (570; 670; 720a; 720b; 820; 1320; 1441) as the partial response derived from the input data (550; 650) based on the similarity between each candidate partial response and the input data (550; 650), Electronic device (101).

8. In any one of paragraphs 1 to 7, The at least one processor (120) individually and / or collectively causes the electronic device (101) to: Based on the input, the source data is configured to be determined as at least one of at least a portion of internal data stored in the memory (130), at least a portion of external device data stored in another electronic device connected to the electronic device (101), or search data obtainable through a search server. Electronic device (101).

9. In any one of paragraphs 1 to 8, The at least one processor (120) individually and / or collectively causes the electronic device (101) to: Based on the query (540; 640; 710a; 710b; 810; 1310; 1401) querying the user's private information, the source data is configured to be determined as internal data stored in the memory (130) of the electronic device (101). Electronic device (101).

10. In any one of paragraphs 1 to 9, The at least one processor (120) individually and / or collectively causes the electronic device (101) to: Obtain summary data of internal data stored in the above memory (130), Based on obtaining the above query (540; 640; 710a; 710b; 810; 1310; 1401), using the summary data and the above query (540; 640; 710a; 710b; 810; 1310; 1401), determining whether the above query (540; 640; 710a; 710b; 810; 1310; 1401) is answerable with the internal data, Based on the above query (540; 640; 710a; 710b; 810; 1310; 1401) being answerable by the internal data, specifying the source data including the internal data, Based on the above query (540; 640; 710a; 710b; 810; 1310; 1401) not being answerable by the internal data, the source data is configured to include at least one of external device data or search data. Electronic device (101).

11. In a method performed by an electronic device, An operation (310) of extracting a plurality of pieces of candidate data related to the query from among the source data based on obtaining a query for querying the source data; An operation (320) of selecting input data among the plurality of candidate data extracted above based on the contents of the plurality of candidate data; An operation (330) of generating a response to the query by applying the query and the selected input data to a generative model; An operation (340) of determining a partial response derived from the selected input data among the above responses; and An operation (350) comprising displaying a visual representation indicating information about the selected input data in an area corresponding to the determined partial response through a display; method.

12. In paragraph 11, The visual representation shown above is, Indicates at least one of an application used to obtain or process the input data, a directory where the input data is stored, a connection path for accessing a page containing the input data, or an external device that shared the input data with the electronic device. method.

13. In any one of paragraphs 11 to 12, The operation (320) of selecting the above input data is: An operation of selecting one of the first candidate data or the second candidate data as the input data based on the content of the first candidate data and the content of the second candidate data being the same, The action (350) of displaying the above visual representation is: Including an action of displaying a visual representation indicating information about the first candidate data and a visual representation indicating information about the second candidate data in an area corresponding to the partial response. method.

14. In any one of paragraphs 11 to 13, The operation (320) of selecting the above input data is: An operation including selecting, as input data, candidate data that is acquired or processed more recently than other candidate data among the first candidate data and the second candidate data, based on the difference between the contents of the first candidate data and the contents of the second candidate data. method.

15. A computer-readable recording medium storing one or more computer programs including commands for performing the method of any one of claims 11 to 14.

Images