Electronic device and method for providing search assistant, and non-transitory computer-readable storage medium

Abstract

The present disclosure provides an electronic device. The electronic device can: activate a search assistant during execution of a first application; in response to receiving a first user input for a first object on a first full screen, acquire object information and / or attribute information of the first object as a first search condition; display a search box including the first search condition; have at least one user input request execution of a second application; activate the search assistant during execution of the second application; in response to receiving a second user input for a second object included in a second full screen, acquire object information and / or attribute information of the second object as a second search condition; and display the search box including the first search condition and the second search condition.

Description

Electronic device, method, and non-transitory computer-readable storage medium providing a search assistant

[0001] The present disclosure relates to an electronic device, a method, and a non-transitory computer-readable storage medium for providing a search assistant.

[0002] When you enter a text or image-based search term (query) using a search engine, you can use the search results to find information about the search term from various information sources such as the web, databases, and documents.

[0003] For image-based search services, similar images can be searched by analyzing information in the image, visual information in the image can be analyzed to search the analysis results, or objects, text, or scenes included in the image can be recognized to provide related search results.

[0004] Additionally, with recent advancements in artificial intelligence, natural language processing using LLM (large language model) and NLP (natural language processing) is enabling more specific and detailed searches than keyword-based search results.

[0005] The above information may be provided as background art to aid in understanding the present disclosure. No claim or determination is made as to whether any of the above is applicable as prior art related to the present disclosure.

[0006] An electronic device according to one embodiment of the present disclosure includes a display; a communication circuit; a memory; and at least one processor including a processing circuit; The memory, when individually or collectively executed by the at least one processor, causes the electronic device to: in response to receiving a search assistant activation input during execution of a first application, activate a search assistant to wait for a user input for selecting a search target for the entire screen output through the display; in response to receiving a first touch input for a first object of the first entire screen output through the display by execution of the first application, acquire at least one of object information or attribute information of the first object as a first search condition according to the type of the first touch input, and display a search box including the first search condition by overlapping it on a part of the first entire screen; in response to receiving a search assistant activation input during execution of a second application requested by at least one user input, activate the search assistant to wait for a user input for selecting a search target for the entire screen output through the display; and In response to receiving a second touch input for a second object included in a second full screen output through the display by execution of an application, instructions may be stored that cause at least one of object information or attribute information of the second object to be acquired as the second search condition according to the type of the second touch input, and to overlap and display the search box including the first search condition and the second search condition on a part of the second full screen.

[0007] According to one embodiment, a method of an electronic device may be provided. The method may activate a search assistant to wait for a user input for selecting a search target for an entire screen output through a display in response to receiving a search assistant activation input while a first application is being executed. The method may obtain, as a first search condition, at least one of object information or attribute information of the first object according to a type of the first touch input, in response to receiving a first touch input for a first object on a first entire screen output through the display by executing the first application. The method may display a search box including the first search condition by overlapping a portion of the first entire screen. The method may activate the search assistant to wait for a user input for selecting a search target for an entire screen output through the display in response to receiving a search assistant activation input while executing a second application, when execution of the second application is requested by at least one user input. The method may, in response to receiving a second touch input for a second object included in a second full screen output through the display by executing the second application, acquire at least one of object information or attribute information of the second object as the second search condition according to the type of the second touch input. The method may display the search box including the first search condition and the second search condition by overlapping it on a portion of the second full screen.

[0008] According to one embodiment, a storage medium storing at least one computer-readable instruction may be provided. The at least one instruction, when executed by at least a part of at least one processor of an electronic device, may cause the electronic device to perform at least one operation. The at least one operation may, in response to receiving a search assistant activation input during execution of a first application, activate a search assistant to wait for a user input for selecting a search target for an entire screen output through the display. The at least one operation may, in response to receiving a first touch input for a first object on a first entire screen output through the display by execution of the first application, acquire at least one of object information or attribute information of the first object as a first search condition depending on the type of the first touch input. The at least one operation may display a search box including the first search condition by overlapping it on a portion of the first entire screen. The at least one operation may activate the search assistant to wait for a user input for selecting a search target for the entire screen output through the display in response to at least one user input requesting execution of a second application and receiving a search assistant activation input during execution of the second application. The at least one operation may acquire at least one of object information or attribute information of the second object as the second search condition in response to receiving a second touch input for a second object included in the second entire screen output through the display by execution of the second application, depending on the type of the second touch input.The at least one action may display the search box including the first search condition and the second search condition by overlapping it on a part of the second full screen.

[0009] An electronic device according to another embodiment of the present disclosure comprises: a display; a communication circuit; a memory; and at least one processor including a processing circuit; The memory may store instructions that, when executed individually or collectively by the at least one processor, cause the electronic device to: activate a search assistant in response to receiving a search assistant activation input while outputting a first screen through the display; identify a type of the first touch input in response to receiving a first touch input for a first area of ​​the first screen while the search assistant is activated; acquire a first search condition for the first area of ​​the first screen according to the type of the first touch input, add the first touch input to a search box, and display a pop-up window displaying the search box and the search conditions included in the search box so as to overlap a portion of the first screen; transmit the search conditions included in the search box as a query to a search assistant server via the communication circuit; and display a search result received from the search assistant server so as to overlap a portion of the first screen via the communication circuit.

[0010] According to one embodiment, a method of an electronic device may be provided. The method may activate a search assistant in response to receiving a search assistant activation input while outputting a first screen through a display of the electronic device. The method may identify a type of a first touch input received on a first area of ​​the first screen while the search assistant is activated. The method may acquire a first search condition for the first area of ​​the first screen based on the type of the first touch input and add it to a search box. The method may display a pop-up window displaying the search box and the search conditions included in the search box by overlapping a portion of the first screen. The method may transmit the search conditions included in the search box as a query to a search assistant server via the communication circuit. The method may display a search result received from the search assistant server by overlapping a portion of the first screen via the communication circuit.

[0011] According to one embodiment, a storage medium storing at least one computer-readable instruction may be provided. The at least one instruction, when executed by at least a part of at least one processor of an electronic device, may cause the electronic device to perform at least one operation. The at least one operation may activate a search assistant in response to receiving a search assistant activation input while outputting a first screen through a display of the electronic device. The at least one operation may identify a type of a first touch input in response to receiving a first touch input on a first area of ​​the first screen while the search assistant is activated. The method may acquire a first search condition for the first area of ​​the first screen according to the type of the first touch input and add it to a search box. The at least one operation may display a pop-up window displaying the search box and search conditions included in the search box by overlapping a portion of the first screen. At least one of the above actions may transmit search conditions included in the search box as a query to a search assistant server via the communication circuit. At least one of the above actions may display search results received from the search assistant server via the communication circuit by overlapping them on another part of the first screen.

[0012] In connection with the description of the drawings, the same or similar reference numerals may be used for the same or similar components.

[0013] FIG. 1 is a block diagram of an electronic device within a network environment according to various embodiments.

[0014] FIG. 2 is a block diagram of an electronic device and a search assistant server according to an embodiment of the present disclosure.

[0015] FIG. 3 is a flowchart illustrating an operation of an electronic device acquiring multiple search conditions according to an embodiment of the present disclosure.

[0016] FIG. 4 is a flowchart illustrating an operation of an electronic device identifying a search condition according to a type of touch input, according to an embodiment of the present disclosure.

[0017] FIG. 5 is a flowchart illustrating an operation of an electronic device receiving a voice signal and obtaining a search condition according to an embodiment of the present disclosure.

[0018] FIG. 6 is an example of an electronic device activating a search assistant according to one embodiment of the present disclosure.

[0019] FIG. 7 is an example of an electronic device obtaining search conditions using a search assistant according to an embodiment of the present disclosure.

[0020] FIG. 8 is an example of an electronic device selecting a search box according to an embodiment of the present disclosure.

[0021] FIGS. 9A and 9B are examples of an operation of an electronic device using a search assistant to store search conditions in a new search box, according to an embodiment of the present disclosure.

[0022] FIG. 10 is an example of an operation of an electronic device obtaining detailed attribute information according to an embodiment of the present disclosure.

[0023] FIG. 11 is a flowchart illustrating an operation of an electronic device performing a search with search conditions within a search box according to an embodiment of the present disclosure.

[0024] FIG. 12 is an example of an electronic device generating a search query according to one embodiment of the present disclosure.

[0025] FIG. 13 is an example of an operation of obtaining multiple search conditions through a multi-display screen of an electronic device according to one embodiment of the present disclosure.

[0026] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. In connection with the description of the drawings, the same or similar reference numerals may be used for identical or similar components. Furthermore, in the drawings and related descriptions, descriptions of well-known functions and configurations may be omitted for clarity and conciseness.

[0027] An embodiment of the present disclosure is described below with reference to the attached drawings.

[0028] FIG. 1 is a block diagram of an electronic device within a network environment according to various embodiments.

[0029] Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of an electronic device (104) or a server (108) via a second network (199) (e.g., a long-range wireless communication network). In one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0030] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or operations. According to one embodiment, as at least a part of the data processing or operations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or an auxiliary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together with the main processor (121). For example, when the electronic device (101) includes the main processor (121) and the auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a given function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0031] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0032] The number of processors (120) may be one or more. For example, the processor (120) may have a multi-core processor structure such as a dual core, quad core, or hexa core.

[0033] The processor (120) can control the operations of the electronic device (101) by executing instructions stored in the memory (130). For example, the processor (120) can correspond to multiple processors that divide multiple operations among the processors and perform them as a whole.

[0034] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0035] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0036] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0037] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0038] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0039] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0040] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0041] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) to an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multi-media interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0042] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0043] The haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. According to one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0044] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0045] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0046] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0047] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0048] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large-scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for realizing eMBB, loss coverage (e.g., 164 dB or less) for realizing mMTC, or U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for realizing URLLC.

[0049] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas, for example, by the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device via the at least one selected antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0050] According to various embodiments, the antenna module (197) may form a mmWave antenna module. In one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

[0051] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0052] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0053] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

[0054] FIG. 2 is a block diagram of an electronic device and a search assistant server according to an embodiment of the present disclosure.

[0055] According to an embodiment, an electronic device (101) may generate a query including one or more search conditions and request a search to a search assistant server (201). The electronic device (101) may provide a search assistant that may generate a search query having multiple conditions (e.g., image, text, voice) and multiple properties (e.g., keyword, shape, color, material, object) based on an intuitive user input corresponding to the user's intention. The search assistant may store one or more search conditions, and generate a query for search by combining the multiple search conditions using at least one of LLM, ASR (automatic speech recognition), or NLP. For example, the search assistant may recognize a user's utterance through ASR and convert it into text, extract multiple search conditions from the text using LLM, and the search assistant may use the query generated by combining the multiple search conditions to provide the search results expected by the user.

[0056] An electronic device (101) according to one embodiment may include at least a part of a search assistant request unit (1011), a user input processing unit (1012), a screen analysis unit (1013), a query generation unit (1014), a database (1015), or a UI (user interface) control unit (1016).

[0057] A search assistant server (201) according to one embodiment may include at least a part of a search assistant processing unit (2011), a large language model (LLM) (2012), a large vision model (LVM) / large multimode model (LMM) (2013), an automatic speech recognition (ASR) (2014), a natural language processing (NLP) (2015), or an information retrieval unit (2016).

[0058] LLM refers to a language model based on an artificial neural network that has learned a large amount of text data through pre-training. LLMs can contain significantly more parameters (e.g., over 10 billion) than conventional language models. LLMs can utilize a transformer artificial neural network structure based on the attention mechanism.

[0059] The attention mechanism is a technology that helps artificial intelligence models focus (attention) on important parts of input data. The attention mechanism can be used to predict output data by predicting the degree to which a portion of time-series input data (e.g., input data such as voice or video, or input data of a neural network layer) contributes to the intermediate or final output of the neural network. Recurrent neural networks (RNNs), which sequentially process each element of a sequence, exhibit poor prediction performance when there is information dependency between long time-series distances. However, the attention mechanism can consider information dependency between long time-series distances by controlling the degree of weight concentration (attention) within the overall (or partial) context of the input data.

[0060] A Transformer can be structured as an encoder-decoder. The encoder processes input data and outputs compressed information (e.g., a contextual representation), and the decoder processes the compressed information and outputs token-based data. Each encoder and decoder can include an independent attention network, and a cross-attention network connecting the encoder and decoder can also be included.

[0061] For example, LLM learning may involve pre-training and / or fine-tuning. Pre-training is the process of training the LLM to acquire general linguistic knowledge using large amounts of text data. For example, this may involve self-supervised learning, where the LLM predicts the next word using the previous word sequence in the text string. Fine-tuning is the process of training the LLM to be suitable for a specific domain (e.g., chatbot, translation, summarization, Q&A) or task. The LLM may be further supervised (or adaptively trained) using a dataset tailored to the domain's purpose based on the pre-trained model. The LLM can perform tasks with text inputs containing natural language, called prompts.

[0062] For example, fine-tuning can be omitted during LLM learning. The user can control the prompts provided to the LLM to enhance its performance on the desired task. Similar to in-context learning or zero-shot / few-shot learning, prompts can be supplemented with examples of the task and / or guidance for performing the task. Publicly available LLMs include BERT (Bidirectional Encoder Representations from Transformer) and GPT (generative pre-trained transformer).

[0063] In addition to text, LLM can also receive additional inputs, including visual information (including video). The visual information can be converted into text through separate preprocessing (e.g., image recognition, scene recognition) and included in the prompt to generate a response. Another example is a video encoder that converts the input image into text-aligned image embeddings. Using the text embeddings corresponding to the input text, a separately trained model (e.g., a large multimodal model) can generate a response.

[0064] The term "LLM" can refer to the language neural network model itself, but it can also refer to the model of an LLM-based application (e.g., chatbot, translation, summarization, text classification, sentence generation). For example, an LLM-based chatbot like ChatGPT or an LLM-based translator can also be referred to as "LLM."

[0065] "LLM" may also include an inference engine utilizing the LLM neural network model. For example, "inputting an input prompt to the LLM" may mean "inputting the input prompt to an inference engine based on the LLM." For example, "the output of the LLM for the input prompt" may mean the output information of the last neural network layer of the LLM (or output information modified through further processing) obtained when the input prompt is input to the LLM-based inference engine.

[0066] According to one embodiment, the search assistant request unit (1011) of the electronic device (101) may request a search for one or more search conditions acquired from one or more non-continuous screens. The search assistant request unit (1011) may generate a query based on a user's input value (e.g., selection of a search target object within an image) and transmit the query to the search assistant server. For example, the query may be a prompt to be transmitted to the LLM, prompt generation, or information required for input to the information retrieval unit (2015). The query may combine various types of input values, such as images, text, audio, and video. The query may include multiple sub-queries. The search assistant request unit (1011) may receive response information for the query from the search assistant server and display the information on the screen.

[0067] According to one embodiment, the user input processing unit (1012) of the electronic device (101) may perform an operation to receive and process a user input (e.g., touch, text, voice, motion, keyboard, mouse, button). For example, the user input processing unit (1012) may recognize a touch pattern (e.g., single touch, double touch, rubbing, area setting) in response to receiving a touch input. An operation corresponding to obtaining a search condition may be set according to the type of user input. For example, in the case of a double touch input, the color of the touch input location on the screen may be set to be obtained as attribute information.

[0068] According to one embodiment, the screen analysis unit (1013) of the electronic device (101) may perform an operation of analyzing / processing screen information currently displayed through the display or a selected image based on a user input. The screen information may include text and an image. For example, the screen analysis unit (1013) may convert the selected image information into text information through optical character recognition (OCR), image recognition / classification, and scene recognition / classification. The screen information, user-selected image, text information, screen analysis information, and analysis information of the user-selected image analyzed by the screen analysis unit (1013) may be used to generate a prompt.

[0069] According to one embodiment, the query generation unit (1014) of the electronic device (101) may generate a query to be requested to the search assistant server (201) based on a user input. The query may be a prompt to be transmitted to the LLM (2012) within the search assistant server (201) or necessary information generated by the prompt.

[0070] According to one embodiment, the database (1015) of the electronic device (101) may serve as a storage for storing new data or recalling previously stored data based on user input. The database (1015) may store a search box including search conditions acquired based on user input, retrieve previously stored search boxes based on activation of a search assistant, and accumulate and store additionally acquired search conditions in the search box. The database (1015) may include one or more search boxes. In one embodiment, a search box is a data set including one or more search conditions, may be named based on user input, and may be created, modified, or deleted based on user input. One or more search conditions and a search box name included in a search box may be included in a query.

[0071] According to one embodiment, the UI control unit (1016) of the electronic device (101) may wait for user input or control a UI (e.g., a search box, a search box list, an attribute information list, a search result screen) to be displayed to the user in a search assistant activated state.

[0072] An electronic device (101) according to one embodiment includes a display (e.g., a display module (160) of FIG. 1); a communication circuit (e.g., a communication module (190) of FIG. 1); a memory (e.g., a memory (130) of FIG. 1); and at least one processor (e.g., a processor (120) of FIG. 1) including a processing circuit;The memory (130), when individually or collectively executed by the at least one processor (120), causes the electronic device (101) to: activate a search assistant to wait for a user input for selecting a search target for the entire screen output through the display in response to receiving a search assistant activation input during execution of a first application; acquire at least one of object information or attribute information of the first object as a first search condition according to a feature of the search target corresponding to the first user input in response to receiving a first user input for a first object of the first entire screen output through the display by execution of the first application; and display a search box including the first search condition in an overlapping manner on a part of the first entire screen; and execute a second application requested by at least one user input, and execute a search assistant activation input when the search assistant activation input is received during execution of the second application. In response, the search assistant can be activated to wait for a user input for selecting a search target for the entire screen output through the display, and in response to receiving a second user input for a second object included in the second entire screen output through the display by execution of the second application, instructions can be stored for obtaining at least one of object information or attribute information of the second object as the second search condition according to a characteristic of the search target corresponding to the second user input, and causing the search box including the first search condition and the second search condition to be overlapped and displayed on a part of the second entire screen.;

[0073] According to one embodiment, the user input for selecting the search target during activation of the search assistant may include at least one of a touch input, a voice input, a text input, a motion input, a keyboard input, a mouse input, a stylus pen input, a biosignal input, or a wearable device input.

[0074] According to one embodiment, the operation of overlapping and displaying a search box including the first search condition on a part of the first full screen of the first application may include the operation of transmitting the first search condition included in the search box as a query to a search server through the communication circuit, receiving a search result from the search server, displaying the search box on at least a part of the first full screen, and outputting a screen including the search result on at least another part of the first full screen.

[0075] According to one embodiment, a search prompt may be generated based on one or more search conditions included in the search box using a large language model, the generated search prompt may be transmitted as a query to a search server, search results may be received from the search server, and the search results may be output through the display.

[0076] In one embodiment, in response to the first user input being identified as a selection of the first object, one or more properties of the first object are determined, a user interface for selecting one of the one or more properties of the first object is displayed on a portion of the first full screen, a selection of a first property of the one or more properties is received through the user interface, and the first property is included in the search box as a third search condition.

[0077] According to one embodiment, in response to obtaining the first search condition or the second search condition by the first user input or the second user input, the first search condition or the second search condition may be included in a most recently used search box among a plurality of search boxes included in the memory.

[0078] According to one embodiment, in response to obtaining the first search condition or the second search condition by the first user input or the second user input, a list of a plurality of search boxes included in the memory is displayed on the first full screen or the second full screen, and the first search condition or the second search condition can be included in the first search box by a user input selecting the first search box from the list.

[0079] According to one embodiment, the search box is stored in memory, and one or more search conditions included in the search box can be modified, deleted, or added by user input.

[0080] According to one embodiment, the display of the search box may be maintained on a third screen switched from the first full screen or the second full screen according to the execution of the first application or the execution of the second application, and the search assistant may be deactivated in response to receiving an exit for the search box by a user input.

[0081] In one embodiment, while the search assistant is activated, the entire screen output through the display may have a blur effect applied to the extent that visibility is maintained.

[0082] According to one embodiment, the search assistant may be deactivated in response to a waiting threshold time elapsed without user input while the search assistant is activated.

[0083] In one embodiment, the second application may be identical to the first application, the first full screen may be different from the second full screen, or the first object or the second object may be different.

[0084] According to another embodiment, an electronic device (101) includes a display (160); a communication circuit (190); a memory (130); and at least one processor (120) including a processing circuit; The memory (130), when individually or collectively executed by the at least one processor (120), causes the electronic device (101) to: in response to receiving a search assistant activation input while outputting a first screen through the display (160), activate the search assistant; while the search assistant is activated, in response to receiving a first user input for a first area of ​​the first screen, identify a characteristic of a search target corresponding to the first user input; acquire a first search condition for the first area of ​​the first screen according to the identified characteristic of the search target, add it to a search box, and display a pop-up window displaying the search box and the search conditions included in the search box by overlapping it on a part of the first screen; transmit the search conditions included in the search box as a query to a search assistant server through the communication circuit; and, through the communication circuit, display a search result received from the search assistant server on the first You can store instructions that cause something to be displayed overlapping another part of the screen.

[0085] According to one embodiment, the first user input may include at least one of a touch input, a voice input, a text input, a motion input, a keyboard input, a mouse input, a stylus pen input, a biosignal input, or a wearable device input.

[0086] According to one embodiment, the search box and the pop-up window displaying the search conditions included in the search box may display a thumbnail, attribute information, and object information for each of the search conditions.

[0087] In one embodiment, in response to activating the search assistant, a list including one or more search boxes stored in the memory, a recent search box, or a new search box is called, and in response to the recent search box or the new search box being called, the first search condition is added to the called recent search box or the new search box, and in response to the list including the one or more search boxes being called, the first search condition is added to the first search box according to a user input selecting a first search box from the list.

[0088] According to one embodiment, the search box is stored in memory, and one or more search conditions included in the search box can be modified, deleted, or added by user input.

[0089] According to one embodiment, a search prompt may be generated based on one or more search conditions included in the search box using a large language model, the generated search prompt may be transmitted as a query to the search assistant server, search results may be received from the search assistant server, and the search results may be output through the display.

[0090] According to one embodiment, in response to the first user input being identified as selection of an object in the first area, one or more properties for the object are determined, a user interface for selecting one of the one or more properties for the object is displayed on a portion of the first screen, a selection of a first property of the one or more properties is received through the user interface, and the first property is included in the search box as a second search condition.

[0091] According to one embodiment, the display of the search box may be maintained on a third screen switched from the first full screen or the second full screen according to the execution of the first application or the execution of the second application, and the search assistant may be deactivated in response to receiving an exit for the search box by a user input.

[0092] According to one embodiment, while the search assistant is activated, the search assistant may receive voice, text, motion, keyboard, or mouse input, and obtain at least one of object information or object attribute information extracted based on the received voice, text, motion, keyboard, or mouse input as a third search condition, and include the third search condition in the search box.

[0093] According to an embodiment, a search assistant processing unit (2011) of a search assistant server (201) may process a query (e.g., a search word, a prompt, or information required to generate a prompt) received from a search assistant request unit (1011) to search / generate required information and transmit the search / generate the required information to an electronic device (101). The search assistant processing unit (2011) may utilize at least some of LLM (2012), LVM / LMM (2013), ASR (2014), NLP (2015), or information retrieval unit (2016) for information retrieval. For example, the search assistant processing unit (2011) may generate a response using at least one of LLM (2012), LVM / LMM (2013), ASR (2014), or NLP (2015) depending on the data type of the received query.

[0094] According to one embodiment, the large language model (LLM) (2012) of the search assistant server (201) can receive a text query and generate a response. The large vision model (LVM) / large multimode model (LMM) (2013) can receive an image query or a query composed of text and image and generate a response. The LVM can learn a large amount of images, videos, or other visual information to perform various computer vision tasks. The LMM is a large-scale artificial intelligence model that can simultaneously process data in different modalities such as text, images, and videos. The query may include a prompt. The output of the LLM (2012) or LVM / LMM (2013) may include an action to be performed. The action to be performed may include an additional search through the information retrieval unit (2016).

[0095] In one embodiment, LLM refers to an artificial neural network-based language model that has learned a large amount of text data through pre-training. LLM may contain significantly more parameters (e.g., over 10 billion) than conventional language models. LLM may utilize a transformer artificial neural network structure based on an attention mechanism.

[0096] The attention mechanism is a technique that helps artificial intelligence models focus (attention) on important parts of input data. The attention mechanism can be used to predict output data by predicting the degree to which a portion of time-series input data (e.g., voice or video input data, or input data from a layer of a neural network) contributes to the intermediate or final output of the neural network. Recurrent neural networks (RNNs), which sequentially process each element of a sequence, exhibit poor prediction performance when there is information dependency between long time-series distances. However, the attention mechanism can consider information dependency between long time-series distances by controlling the degree of weight concentration (attention) within the overall (or partial) context of the input data.

[0097] A Transformer can be structured as an encoder-decoder. The encoder processes input data and outputs compressed information (e.g., a contextual representation), while the decoder processes the compressed information and outputs token-based data. Each encoder and decoder can include an independent attention network, or a cross-attention network connecting the encoder and decoder.

[0098] For example, LLM learning may involve pre-training and / or fine-tuning. Pre-training involves training the LLM to acquire general linguistic knowledge using large amounts of text data. For example, this may involve self-supervised learning, where the LLM predicts the next word based on the previous word sequence in a text string. Fine-tuning involves training the LLM to be suitable for a specific domain (e.g., chatbot, translation, summarization, Q&A) or task. The LLM can be further supervised (or adaptively trained) using a dataset tailored to the domain's purpose, based on the pre-trained model. The LLM can perform tasks using text inputs containing natural language, called prompts. For example, fine-tuning may be omitted during LLM learning. The user can control the prompts provided to the LLM to improve the performance of a desired task. Examples of tasks and / or guidance for performing the task can be added to the prompts, similar to in-context learning or zero-shot / few-shot learning. As a published LLM (2012), there are BERT (Bidirectional Encoder Representations from Transformer) and GPT (generative pre-trained transformer).

[0099] The term "LLM" can refer to the language neural network model itself, but it can also refer to the model of an LLM-based application (e.g., chatbot, translation, summarization, text classification, sentence generation). For example, an LLM-based chatbot like ChatGPT or an LLM-based translator can also be referred to as "LLM."

[0100] "LLM" may also include an inference engine utilizing the LLM neural network model. For example, "inputting an input prompt to the LLM" may mean "inputting the input prompt to an inference engine based on the LLM." For example, "the output of the LLM for the input prompt" may mean the output information of the last neural network layer of the LLM (or output information modified through further processing) obtained when the input prompt is input to the LLM-based inference engine.

[0101] According to one embodiment, the LVM / LMM (2013) of the search assistant server (201) can analyze a query and generate a response when an image query or a query combining an image and text is input.

[0102] According to one embodiment, the automatic speech recognition (ASR) (2014) of the search assistant server (201) can convert a query containing a voice signal into text through automatic speech recognition.

[0103] According to one embodiment, the natural language processing (NLP) (2015) of the search assistant server (201) can process text when the query includes text to facilitate the generation of a response in the LLM (2012) or information retrieval unit (2015).

[0104] According to one embodiment, the information retrieval unit (2016) of the search assistant server (201) can obtain search results for a query by searching for the query based on an internal database or an external database. The database can be implemented in various types. For example, the database can be SQL (structured query language) or NoSQL (not only SQL). The information retrieval unit (2016) can transmit search result information for the query to the electronic device (101) via the server assistant processing unit (2011).

[0105] FIG. 3 is a flowchart illustrating an operation of an electronic device acquiring multiple search conditions according to an embodiment of the present disclosure.

[0106] According to one embodiment, an electronic device (101) can acquire multiple search conditions from non-sequential screens and generate a query combining the multiple search conditions. In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

[0107] According to one embodiment, operations 310 to 380 may be understood to be performed in a processor (e.g., processor (120) of FIG. 1) of an electronic device (e.g., electronic device (101) of FIG. 1 and / or FIG. 2).

[0108] In operation 310, an electronic device (101) according to an embodiment may display a first screen according to execution of a first application through a display (e.g., a display module (160) of FIG. 1). The first application may be a variety of applications, such as a web application, a gallery application, a camera application, or a video content providing application. The first screen may be displayed on the entire display.

[0109] In operation 320, an electronic device (101) according to an embodiment may receive a search assistant activation input while a first screen is displayed, thereby activating a search assistant. The search assistant may obtain various types of user inputs as search conditions and provide a search box to store a plurality of search conditions accumulated from non-consecutive screens. The search assistant may receive at least some of touch, voice, text, motion, keyboard, mouse, or key button inputs.

[0110] In one embodiment, the search assistant activation input may be a specified user input value. For example, the search assistant activation input may be set to a long-touch input on the lower portion of the display. Alternatively, the search assistant activation input may be set to a touch input drawing the letter S anywhere on the display.

[0111] In one embodiment, when the search assistant is activated, it may be in a state of waiting for a user input to select a search target from the entire screen output through the display. In one embodiment, the user input may include at least one of a touch input, a voice input, a text input, a motion input, a keyboard input, a mouse input, a stylus pen input, a biosignal input (e.g., gaze, movement), or a wearable device input. For example, while the search assistant is in an activated state, in response to receiving a touch input for a first object on the first screen, the touch input for the first object may be analyzed to obtain a search condition.

[0112] In operation 330, an electronic device (101) according to an embodiment may receive a first user input for a first object on a first screen during activation of a search assistant, and obtain a first search condition for the first object according to the first touch input.

[0113] The electronic device (101) can identify the characteristics of a search target based on the type of user input and determine the search condition. The characteristics of the search target may be object information or attribute information of the object corresponding to the search target. In the case of a touch input, the electronic device (101) can identify the characteristics of the search target based on the type of the touch input pattern. The touch pattern may include a left swipe, a right swipe, a single touch, a double touch, a circle (including an irregular circle), an irregular shape, a two-finger touch, a zoom-in touch, or a zoom-out touch. The electronic device (101) can recognize at least one of a touch area, an object included in the touch area, and an attribute of the object as a search condition based on the touch pattern. The electronic device (101) can analyze the touch pattern of the first user input to obtain a first search condition for a first object corresponding to the touch pattern. According to one embodiment, the first object may be image information for an area corresponding to the coordinates (or the surroundings) of the first touch input among the first screen being displayed. According to one embodiment, the first object may be information about an image object corresponding to the coordinates (or the vicinity thereof) of the first touch input. The image object may be determined based on image processing such as image recognition, pattern recognition, and scene recognition. The first search condition may include at least one of object information or attribute information of the first object. For example, the search condition may include identified object information (e.g., car, clothes, sofa, dog) and attribute information (e.g., color, material, shape). Depending on the first user input, the first search condition may be object information or attribute information, and may be object information and attribute information about the object.

[0114] In operation 340, an electronic device (101) according to an embodiment may display a search box including a first search condition by overlapping a portion of a first screen. The electronic device (101) may display the search box as a pop-up window on a portion of the first screen. The search box may include the first search condition and one or more previously stored search conditions. The search box may display a thumbnail, attribute information, and object information for each search condition.

[0115] In operation 350, the electronic device (101) according to an embodiment may output a second screen through the display according to the execution of the second application. The state of the electronic device (101) according to operation 350 may be a state in which a first screen according to the first application is displayed, a request for execution of the second application is received by at least one user input, and one or more screens are displayed according to the execution of the second application, and the second screen is displayed. The first screen and the second screen may be temporally non-contiguous, and the first application and the second application may be different applications. According to an embodiment, the electronic device (101) may maintain or terminate the display of the search box when the screen switches according to the execution of the same application or when the screen switches according to a change of application. The electronic device (101) may maintain the activation of the search assistant while the search box is displayed.

[0116] In operation 360, the electronic device (101) according to one embodiment may receive a search assistant activation input while displaying a second screen according to execution of a second application, and activate the search assistant. The operation may correspond to at least a part of operation 320. The electronic device (101) may receive a search assistant activation input while displaying any screen, and may activate the search assistant to wait for a user input for obtaining a search condition. In one embodiment, the user input may include at least one of a touch input, a voice input, a text input, a motion input, a keyboard input, a mouse input, a stylus pen input, a biosignal input, or a wearable device input. According to one embodiment, the electronic device (101) may display a search box including a first search condition by overlapping at least a part of the second screen according to the search assistant activation. The search box may be movable within the second screen according to the user input.

[0117] According to an embodiment, the electronic device (101) may maintain the search box that was activated on the previous screen to be displayed on the second screen while the screen transitions from the first screen to the second screen, and may maintain the search assistant activation while the search box is displayed so that a separate search assistant activation input is not received. In this case, the 360 ​​operation may not be performed. Alternatively, according to an embodiment, the electronic device (101) may stop displaying the search box when there is no additional search condition input on the first screen or when the screen transitions. When the electronic device (101) receives a search assistant activation input, the search box may be displayed again on the current screen, and the search box may be displayed including the previously added search condition.

[0118] In operation 370, the electronic device (101) according to one embodiment may, in response to receiving a second user input for a second object on a second screen while the search assistant is activated, obtain a second search condition for the second object based on the second user input. This operation may correspond to at least a portion of operation 330. The operation of analyzing the second touch input and determining the search condition is described in detail in FIG. 4.

[0119] In operation 380, an electronic device (101) according to an embodiment may display a search box including a first search condition and a second search condition in the form of a pop-up window on at least a portion of the current screen (second screen).

[0120] FIG. 4 is a flowchart illustrating an operation of an electronic device identifying a search condition according to a type of touch input, according to an embodiment of the present disclosure.

[0121] According to an embodiment, an electronic device (101) can receive a touch input as a user input. The electronic device (101) can identify the type of touch input (e.g., a pattern) and obtain a corresponding designated search condition. For example, a double touch input for a first point can be set to obtain the color of the first point as attribute information. In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

[0122] According to one embodiment, operations 410 to 490 may be understood to be performed in a processor (e.g., processor (120) of FIG. 1) of an electronic device (e.g., electronic device (101) of FIG. 1 and / or FIG. 2).

[0123] In operation 410, an electronic device (101) according to one embodiment may receive a first touch input for a current screen output through a display.

[0124] In operation 420, the electronic device (101) according to one embodiment may determine whether the type of the first touch input corresponds to object selection. In one embodiment, a circle touch may be set to select an object displayed in the corresponding part. The circle touch may indicate a circular touch pattern.

[0125] In operation 430, the electronic device (101) according to one embodiment can identify an object in an area corresponding to the first touch input in response to determining that the first touch input corresponds to object selection. The electronic device (101) can identify an object corresponding to the area corresponding to the first touch input using image recognition technology. The object can be, for example, a person, an animal, or an object.

[0126] In operation 440, an electronic device (101) according to an embodiment may display a list of attribute candidates for an identified object. The electronic device (101) may identify one or more attributes for the identified object. For example, if the object is a person, one or more attributes that a person may have may be predefined, such as adult, child, female, male, race, height, or age. If the identified object is 'clothes', the electronic device (101) may identify 'shape, color, texture (material), season, and pattern' as attributes that clothes may have. The electronic device (101) may display a list of one or more attributes as a user interface for selecting one of the one or more attributes for the identified object. For example, the electronic device (101) may display a list of 'shape, color, texture (material), season, and pattern' in the 'clothes' section of the current screen. As another example, if the identified object corresponds to an 'electronic product', the object's properties may be 'shape, color, size, function, manufacturer'. The electronic device (101) according to one embodiment may recognize one or more objects included in the current screen through technologies such as image recognition and scene recognition, and identify the objects included in the area of ​​the first touch input.

[0127] In operation 450, the electronic device (101) according to one embodiment may determine which of the attribute candidate lists displayed on the first screen is selected by a user input. For example, the electronic device (101) may receive a touch input for selecting 'color' from the attribute list. Alternatively, the electronic device (101) may determine that attribute information has not been selected if it receives a touch input for an area other than the area where the attribute list is displayed on the current screen, or if a certain period of time passes without a user input.

[0128] In operation 451, the electronic device (101) according to one embodiment may acquire object information about the identified object as a search condition in response to none of the attribute candidate list being selected. The electronic device (101) may store 'clothes' as a search condition and may further include detailed information about 'clothes' in the search condition by analyzing the first screen. For example, the electronic device (101) may store 'long-sleeved men's sweater', which embodies the upper concept of 'clothes', as a search condition.

[0129] In operation 452, the electronic device (101) according to one embodiment may, in response to selecting a first attribute from a list of attribute candidates, identify attribute information corresponding to the first attribute and add the attribute information to a search condition. The electronic device (10) may add object information about the identified object and attribute information corresponding to the first attribute together to the search condition. For example, in response to selecting 'color', the electronic device (101) may analyze the current screen to obtain color information (e.g., mint color) of 'clothes'. The electronic device (101) may store both 'long-sleeved men's top sweater' and 'mint color' as object information as search conditions. The operation of determining the attribute value of an object (providing sub-object information of a superior object) according to the type of user touch input of the electronic device (101) can shorten the segmentation preprocessing time and ultimately shorten the search time because the screen analysis or object recognition process is unnecessary.

[0130] In operation 460, in response to determining that the first touch input does not correspond to an object selection, the electronic device (101) according to one embodiment may determine whether the type of the first touch input corresponds to an attribute selection. In one embodiment, a double touch may be set to obtain color information as an attribute value. If the first touch input is neither an object selection nor an attribute selection, the electronic device (101) may not perform any operation for the first touch input.

[0131] In operation 470, the electronic device (101) according to one embodiment may determine an attribute by identifying corresponding attribute information in response to determining that the type of the first touch input corresponds to attribute selection. The electronic device (101) may determine attribute information corresponding to the type of the first touch input. For example, if the first touch input is a rubbing touch, this may correspond to acquiring 'material' information.

[0132] In operation 480, the electronic device (101) according to one embodiment can analyze the current screen to determine attribute information and store the attribute information as a search condition. For example, if the first touch input is to obtain 'material' information, the electronic device (101) can identify a portion of the current screen corresponding to the first touch input through image recognition and scene recognition technology and obtain material information of the corresponding portion. For example, the electronic device (101) can analyze the portion corresponding to the first touch input and obtain (store) attribute information of 'soft fiber texture' as a search condition.

[0133] In one embodiment, the electronic device (101) can determine whether to add the newly acquired attribute information if the acquired attribute information corresponds to the same classification as the attribute information previously stored in the search box. The electronic device (101) can display a user interface on a pop-up screen that allows the user to select whether to replace the newly acquired attribute information or add it as a combined condition. For example, if the user acquires a first texture by a rubbing touch input and then acquires a second texture by the same rubbing touch input on the next screen, the electronic device (101) can display the first texture and the second texture together on the pop-up screen and allow the user to select to delete or add.

[0134] In operation 490, the electronic device (101) according to an embodiment may display a search box indicating search conditions by overlapping a portion of the current screen. The electronic device (101) may display a search box indicating 'long-sleeved men's top sweater' on the current screen by operation 451. The electronic device (101) may display a search box indicating 'long-sleeved men's top sweater' and 'mint color' on the current screen by operation 452. The electronic device (101) may display a search box indicating 'soft fabric texture' on the current screen by operation 480. The search box may display thumbnails for object information and attribute information together, and may display the type of information, whether it is object information or attribute information. When a search condition is clicked, the search box may provide a link to a screen where the search condition is obtained.

[0135] FIG. 5 is a flowchart illustrating an operation of an electronic device receiving a voice signal and obtaining a search condition according to an embodiment of the present disclosure.

[0136] According to an embodiment, an electronic device (101) may receive various types of user input while a search assistant is activated, thereby obtaining search conditions. In an embodiment, the user input may include at least some of touch, text, voice, motion, keyboard, mouse, or key buttons. The electronic device (101) may analyze the user input value, convert it into text, and store it as a search condition. In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

[0137] According to one embodiment, operations 510 to 550 may be understood to be performed in a processor (e.g., processor (120) of FIG. 1) of an electronic device (e.g., electronic device (101) of FIG. 1 and / or FIG. 2).

[0138] In operation 510, the electronic device (101) according to one embodiment may display a first screen through a display. The operation may correspond to at least a part of operation 310 or operation 350 of FIG. 3.

[0139] In operation 520, the electronic device (101) according to one embodiment may activate the search assistant in response to receiving a search assistant activation input. The search assistant may receive user input using one or more input devices. For example, the search assistant may receive user input values ​​through a display, a touchpad, a microphone, an inertial sensor, an acceleration sensor, a keyboard, a mouse, and key buttons. In response to the search assistant being activated, the electronic device (101) may detect user input input through the touchpad, or activate the microphone to detect a voice signal. The operation may correspond to at least a portion of operation 320 or 360 of FIG. 3.

[0140] In operation 530, the electronic device (101) according to one embodiment may receive a voice signal using a microphone while the search assistant is activated. The voice signal may be a user utterance. For example, the electronic device (101) may receive a voice such as "The female protagonist's bag is pretty" as a user utterance through the microphone while the search assistant is activated. In one embodiment, the electronic device (101) may receive a touch input and a voice input simultaneously while the search assistant is activated. For example, the electronic device (101) may receive a circle touch input for "clothes with a mixture of mint and brown" displayed on the screen, and receive a user utterance of "I like that color." The electronic device (101) may extract the color of the circle touch input target according to the circle touch input, and obtain an attribute value that is a composite mixture of the two colors as a search condition.

[0141] In operation 540, an electronic device (101) according to an embodiment may convert a received voice signal into text. The electronic device (101) may convert the voice signal into text and analyze the converted text. The electronic device (101) may extract the keywords "female protagonist" and "bag" from the converted text and search for "female protagonist" and "female protagonist's bag" on the first screen.

[0142] In operation 550, an electronic device (101) according to an embodiment can acquire text as a search condition and add it to a search box. The electronic device (101) can store the text, "The female protagonist's bag is pretty," which is a voice signal converted directly into text, along with screen analysis information as a search condition. The electronic device (101) can store the results of analyzing the text and screen as a search condition.

[0143] FIG. 6 is an example of an electronic device activating a search assistant according to one embodiment of the present disclosure.

[0144] According to one embodiment, an electronic device (101) can obtain multiple search conditions using a search assistant on a full screen output through a display. To provide the search assistant, the electronic device (101) can first receive user input for activating the search assistant.

[0145] Referring to FIG. 6, the first screen (610) may be displayed by executing a first application. The first screen may include text, images, and videos. The first application may correspond to any of a variety of types (e.g., a web page, a gallery application, a streaming content provider application). When a user discovers a search target on the first screen (610), the user may perform an action corresponding to user input to activate a search assistant to obtain search conditions.

[0146] The second screen (620) illustrates a touch input (621) generated by the user on the lower portion of the first screen. As shown in FIG. 6, the user input for activating the search assistant may be a touch input (621) on a specific area (the lower portion) of the display. In one embodiment, the specific area of ​​the display may be the home key area. Alternatively, in one embodiment, the user input for activating the search assistant may be a voice command including a specific command. For example, the voice command may be "Hi Bixby Search" or "Hi Bixby Search Box." In response to receiving the touch input (621) on the lower portion of the display, the electronic device (101) may display a third screen (630) in which the search assistant is activated. The third screen (630) may include a blur effect at a level that maintains visibility due to the activation of the search assistant. The electronic device (101) may blur the current screen, display a search box, or display a message indicating that the search assistant is activated (e.g., "Please add search conditions to the search box") to distinguish between an activated and deactivated state of the search assistant. When the current screen is blurred, a search box is displayed, or a message indicating that the search assistant is activated is displayed, the user may perform a specific action (e.g., touch input, voice input) to obtain a search condition. The specific action to obtain a search condition may be a user input type designated to obtain object information or attribute information. The electronic device (101) may analyze the specific action to identify a corresponding search condition obtaining action.

[0147] FIG. 7 is an example of an electronic device obtaining search conditions using a search assistant according to an embodiment of the present disclosure.

[0148] An electronic device (101) according to one embodiment can obtain one or more search conditions by user input while the search assistant is activated.

[0149] In the first screen (710), the electronic device (101) can receive a touch input (711) for selecting a first object.

[0150] In the second screen (720), the electronic device (101) can add the first object (7223) selected according to the touch input (711) to the search box (732) and display the search box (732) on a part of the current screen. The search box (732) can further include previously stored search conditions (7221, 7222) in addition to the object information (7223) added by the touch input (711). The search box (732) can accumulate, store, and display search conditions acquired by the user input. In response to acquiring the first object (7223) as a search condition, the electronic device (101) can generate a query using the search conditions (7221, 7222, 7223) included in the search box (732). The electronic device (101) may display a search window (723) in which search conditions (7221, 7222, 7223) are entered as queries. In one embodiment, a search result (724) may be displayed in response to the determination of the first object. The search result (724) may be the result of an information search performed based on the search conditions (7221 to 7223) within the search box (722).

[0151] In addition, the electronic device (101) can identify the types of attributes that the first object can have according to the touch input (711) and display a candidate attribute list (721) on a portion of the current screen in relation to the first object. In one embodiment, attribute values ​​included in the candidate attribute list can be determined based on the results determined by analyzing an image of the first object. For example, among the attributes designated through image classification implemented with an artificial neural network, n attributes having a high classification score (e.g., classification score, confidence score) can be determined as candidate attributes. For example, the candidate attributes can be sorted according to a high classification score.

[0152] In the third screen (730), the electronic device (101) can select Texture from the candidate property list (721) of the first object and receive a drag-and-drop touch input (7311) to the search box (732). In response to the drag-and-drop touch input (7311), the electronic device (101) can store and display the Texture property (7324) of the first object (7323) as a search condition in the search box (732). As in the example of FIG. 7, the electronic device (101) can obtain two search conditions (7323, 7324) in one screen. In response to obtaining the Texture property (7324) as a search condition, the electronic device (101) can generate a query using the search conditions (7321, 7322, 7323, 7324) included in the search box (732). The electronic device (101) may display a search window (733) in which search conditions (7321, 7322, 7323, 7324) are entered as queries. In one embodiment, a search result (734) may be displayed in response to determining a texture property of a first object. The search result (734) may be a result of an information search performed based on the search conditions (7321 to 7324) within the search box (732).

[0153] In one embodiment, the search window (723) may include a search button for searching. The search button may be, for example, the title display portion of a search box. Alternatively, the search box may include a separate search icon. The electronic device (101) may update and display search results based on search conditions (7321-7324) in response to user input to the search button.

[0154] According to an embodiment, the electronic device (101) can delete each search condition displayed in the search box (732). For example, the electronic device (101) can delete the pattern search condition (7322) by long-pressing the pattern search condition (7322) in the search box (732) and then dragging and dropping it onto the trash can icon when a trash can icon is displayed. In this case, the search condition can be considered permanently deleted from the search box.

[0155] According to one embodiment, the electronic device (101) can delete each search condition displayed in the search window (733). For example, the electronic device (101) can delete a pattern search condition (not shown) by long-pressing the pattern search condition (not shown) in the search window (733) and then dragging and dropping it onto the trash can icon when a trash can icon is displayed. In this case, the search condition can be considered to be excluded only from the current search query, and the pattern search condition (7322) can be maintained in the search box (732).

[0156] FIG. 8 is an example of an electronic device selecting a search box according to an embodiment of the present disclosure.

[0157] According to one embodiment, the electronic device (101) can select one of one or more search boxes stored in the memory and store the currently acquired search conditions.

[0158] An electronic device (101) according to an embodiment may store and display search boxes stored in a memory as a list (810). The search boxes may be identified by a name entered by a user. The search box list (810) may include search boxes corresponding to sofas, cars, dog clothes, and blind date looks. The search box list (810) may also include a 'new' item for creating a new search box. In response to selecting the 'new' item from the search box list (810), the electronic device (101) may create and display a new search box (820). The new search box (820) may be given an arbitrarily designated name, and the name may be changed by user input. In response to selecting the 'blind date look' item from the search box list (710), the electronic device (101) may display a search box displaying search conditions included in the blind date look search box (830). The electronic device (101) can modify or delete search conditions included in the blind date look search box (830) according to user input.

[0159] An electronic device (101) according to an embodiment can store and manage search boxes based on information about the creation time and the last time of calling (referencing) the search box. The electronic device (101) can be set to display the most recently called search box as a default when a search condition is acquired, according to user settings. Alternatively, the electronic device (101) can be set to create and display a new search box when a search condition is acquired, according to user settings. The electronic device (101) can display a list of search boxes stored in a memory when a search condition is acquired, according to user settings, and receive a selection for any one of the search boxes displayed in the list.

[0160] An electronic device (101) according to an embodiment may maintain the display of a search box while a search assistant is activated. For example, after acquiring a search condition and storing it in a search box, the electronic device (101) may display the search box on at least a portion of the screen while executing another app or performing another function to display another screen. The electronic device (101) may display the search box at a size smaller than that displayed upon acquiring the search condition. After displaying the search box upon acquiring the search condition, the electronic device (101) may not display the search box if no user input corresponding to the acquisition of the search condition is received for a predetermined waiting threshold time.

[0161] According to one embodiment, search boxes are stored in memory and can be permanently stored until they are arbitrarily deleted by the user. Search boxes can be stored within the memory limit. The number of search conditions that can be stored in each search box can be determined by a threshold. For example, the number of search conditions can be determined based on the search query size acceptable to the search assistant server. If the storage space is insufficient or the number of search conditions exceeds the threshold, the electronic device (101) can display a message indicating that no more search boxes or search conditions can be added. In addition, the electronic device (101) can display a pop-up message suggesting that the user organize the list of old search boxes to manage the permanently stored search boxes.

[0162] According to one embodiment, when the electronic device (101) receives a user input for selecting each search condition within the search box (830), it may display detailed information about the search condition on a larger screen. For example, in response to a user input for selecting a season search condition, the electronic device (101) may display the original screen from which the season search condition was obtained and the analysis result for the season search condition (e.g., early winter) together.

[0163] FIGS. 9A and 9B are examples of an operation of an electronic device using a search assistant to store search conditions in a new search box, according to an embodiment of the present disclosure.

[0164] An electronic device (101) according to one embodiment may provide a search assistant to a user so that search conditions can be intuitively obtained.

[0165] On the first screen (910), the electronic device (101) can display a web page. When the electronic device (101) receives a search assistant activation input (e.g., a long touch input on the lower area of ​​the display) (621), the electronic device (101) can activate the search assistant.

[0166] In the second screen (920), the electronic device (101) can display a state in which the search assistant is activated, blurring the entire screen to an extent that visibility is maintained.

[0167] In the third screen (930), the electronic device (101) can receive a first touch input for the first object. For example, the first touch input can be a double touch input, and the double touch input can be matched with a color attribute information acquisition operation.

[0168] In response to receiving the first touch input, the electronic device (101) may display a new search box on the fourth screen (940) and add the color search information acquired by the first touch input to the search box and display it. The electronic device (101) may receive a touch input for the name portion of the new search box and perform a name change for the search box. The electronic device (101) may receive the text 'SOFA' as in the fifth screen (950) and set it as the search box name.

[0169] FIG. 10 is an example of an operation of an electronic device obtaining detailed attribute information according to an embodiment of the present disclosure.

[0170] According to one embodiment, the electronic device (101) can receive user input and obtain object information or attribute information as a search condition in a search assistant activated state.

[0171] Referring to FIG. 10, when a first web page (1010) is displayed and a search assistant is activated (1020) by a user input, a first touch input (e.g., double touch) for a first object (e.g., sofa) may be received (1030) to obtain attribute information. In order to obtain more detailed attribute information than the attribute information obtained at the resolution at which the first web page (1010) is displayed, the electronic device (101) may access the address of the first web page (1010) to obtain more detailed attribute information from a high-resolution screen of the original image. Alternatively, the electronic device (101) may increase the resolution of the displayed image by using an image contrast method and obtain accurate attribute information (e.g., texture).

[0172] FIG. 11 is a flowchart illustrating an operation of an electronic device performing a search with search conditions within a search box according to an embodiment of the present disclosure.

[0173] According to one embodiment, the electronic device (101) can perform a search by generating a query using search conditions included in the search box. In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

[0174] According to one embodiment, operations 1110 to 1140 may be understood to be performed in a processor (e.g., processor (120) of FIG. 1) of an electronic device (e.g., electronic device (101) of FIG. 1 and / or FIG. 2).

[0175] In operation 1110, an electronic device (101) according to an embodiment may generate a search prompt based on search conditions within a search box. The electronic device (101) may generate a search prompt based on one or more search conditions included within the search box using LLM.

[0176] In operation 1120, the electronic device (101) according to one embodiment may transmit the generated search prompt as a query to a search assistant server.

[0177] In operation 1130, an electronic device (101) according to an embodiment may receive search results from a search assistant server. The search assistant server may obtain search results for a received query using LLM, NLP, ASR, and an information retrieval unit.

[0178] In operation 1140, an electronic device (101) according to an embodiment may display search results through a display. The electronic device (101) may display a search results screen and a search prompt together in a search window.

[0179] FIG. 12 is an example of an electronic device generating a search query according to one embodiment of the present disclosure.

[0180] An electronic device (101) according to one embodiment can generate a search query by combining a plurality of search conditions.

[0181] Referring to FIG. 12, the first search condition for the first object obtained in the first screen (1201) may be 'Pomeranian'. The electronic device (101) may analyze the first object and the first screen to generate a first search prompt of 'brown Pomeranian with a teddy bear appearance'.

[0182] The second search condition for the second object obtained from the second screen (1202) may be 'rabbit clothes'. The electronic device (101) may analyze the second object and the second screen to generate a second search prompt for 'cute clothes with blue or pink rabbit ear decorations'.

[0183] An electronic device (101) according to an embodiment may display a search box (1203) including a first search condition and a second search condition. In response to a touch input for a name portion of the search box (1203), the electronic device (101) may execute a search function using the search condition within the search box. The electronic device (101) may generate a query and perform a search request based on a first search prompt and a second search prompt generated from a first object and a second object of different characteristics. For example, the electronic device (101) may automatically generate a search query such as 'Search for cute clothes with blue or pink rabbit ears that suit a brown Pomeranian with a teddy bear appearance' based on the first search prompt and the second search prompt.

[0184] According to an embodiment, an electronic device (101) can display a search window (1210) displaying a search query and a search result screen (1230) through a display. The electronic device (101) displays an automatically generated search query together with search results (1230) in the search window (1210), thereby allowing a user to check how the search conditions he or she added were applied in an actual search. The electronic device (101) can display search conditions in a search box and display a search query generated according to the search conditions in the search window, thereby assisting the user in using a search assistant.

[0185] FIG. 13 is an example of an operation of obtaining multiple search conditions through a multi-display screen of an electronic device according to one embodiment of the present disclosure.

[0186] According to one embodiment, when an electronic device (1301) includes multiple displays or displays execution screens by different applications in multiple areas on one display, multiple search conditions can be obtained from the current screen displayed through the display without switching screens.

[0187] Referring to FIG. 13, the electronic device (1301) can obtain a first search condition in response to a first touch input in a first area (1310). The electronic device (1301) can obtain a second search condition in response to a second touch input in a second area (1320). The electronic device (1301) can display a search box (1330) including the first search condition and the second search condition by overlapping at least a portion of the first area (1310) and the second area (1320).

[0188] The embodiments of this document and the terminology used herein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0189] The term "module" used in the embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0190] One embodiment of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0191] According to one embodiment, the method according to one embodiment disclosed in this document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play Store™) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0192] According to one embodiment, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and arranged in other components. According to one embodiment, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to one embodiment, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In electronic devices, display; communication circuit; memory; and At least one processor comprising a processing circuit; The memory, when executed individually or collectively by the at least one processor, causes the electronic device to: In response to receiving a search assistant activation input while the first application is running, activate the search assistant to wait for user input to select a search target for the entire screen output through the display, In response to receiving a first user input for a first object of a first full screen output through the display by executing the first application, at least one of object information or attribute information of the first object is acquired as a first search condition according to a feature of a search target corresponding to the first user input, A search box including the first search condition is displayed by overlapping a part of the first full screen, The execution of a second application is requested by at least one user input, In response to receiving the search assistant activation input while the second application is running, the search assistant is activated to wait for a user input for selecting a search target for the entire screen output through the display, In response to receiving a second user input for a second object included in a second full screen output through the display by executing the second application, at least one of object information or attribute information of the second object is acquired as the second search condition according to the characteristics of the search target corresponding to the second user input, and An electronic device storing instructions that cause the search box including the first search condition and the second search condition to be displayed overlappingly on a portion of the second full screen.

2. In paragraph 1, The memory, when individually or collectively executed by the at least one processor, causes the electronic device to: An electronic device storing commands that cause a user input for selecting a search target during activation of the search assistant to include at least one of a touch input, a voice input, a text input, a motion input, a keyboard input, a mouse input, a stylus pen input, a biosignal input, or a wearable device input.

3. In paragraph 1, The memory, when individually or collectively executed by the at least one processor, causes the electronic device to: An action of overlapping and displaying a search box including the first search condition on a part of the first full screen of the first application is as follows: Through the above communication circuit, the first search condition included in the search box is transmitted as a query to the search server, Receive search results from the above search server, Displaying the search box on at least a portion of the first full screen, and An electronic device storing commands that cause the device to output a screen including the search results to at least another portion of the first full screen.

4. In paragraph 1, The memory, when individually or collectively executed by the at least one processor, causes the electronic device to: Generate a search prompt based on one or more search conditions included in the search box using a large language model, Send the above generated search prompt as a query to the search server, Receive search results from the above search server, and An electronic device storing commands that cause the search results to be output through the display.

5. In paragraph 1, The memory, when individually or collectively executed by the at least one processor, causes the electronic device to: In response to the first user input being identified as a selection for the first object, determining one or more properties for the first object; Displaying a user interface for selecting one or more properties of the first object on a portion of the first full screen; Receiving a selection for a first attribute among said one or more attributes via said user interface, An electronic device storing commands that cause the first property to be included in the search box as a third search condition.

6. In paragraph 1, The memory, when individually or collectively executed by the at least one processor, causes the electronic device to: An electronic device storing instructions that cause the most recently used search box among a plurality of search boxes included in the memory to include the first search condition or the second search condition in response to obtaining the first search condition or the second search condition by the first user input or the second user input.

7. In paragraph 1, The memory, when individually or collectively executed by the at least one processor, causes the electronic device to: In response to obtaining the first search condition or the second search condition by the first user input or the second user input, displaying a list of a plurality of search boxes included in the memory on the first full screen or the second full screen, An electronic device storing commands that cause the first search box to include the first search condition or the second search condition in response to a user input selecting the first search box from the above list.

8. In paragraph 1, The memory, when individually or collectively executed by the at least one processor, causes the electronic device to: The above search box is stored in memory, An electronic device storing commands that cause one or more search conditions included in the search box to be modified, deleted or added by user input.

9. In paragraph 1, The memory, when individually or collectively executed by the at least one processor, causes the electronic device to: Maintain the display of the search box in the third screen switched from the first full screen or the second full screen depending on the execution of the first application or the execution of the second application; An electronic device storing instructions that cause the search assistant to be deactivated in response to receiving an exit from the search box by user input.

10. In paragraph 1, The memory, when individually or collectively executed by the at least one processor, causes the electronic device to: An electronic device storing commands that cause the entire screen output through the display to be blurred to a degree that maintains visibility while the above search assistant is activated.

11. In paragraph 1, The memory, when individually or collectively executed by the at least one processor, causes the electronic device to: An electronic device storing commands that cause the search assistant to be deactivated in response to a waiting threshold time elapsed without user input while the search assistant is activated.

12. In paragraph 1, The memory, when individually or collectively executed by the at least one processor, causes the electronic device to: An electronic device in which the second application is the same as the first application, the first full screen and the second full screen are different, or the first object or the second object is different.

13. In electronic devices, display; communication circuit; memory; and At least one processor comprising a processing circuit; The memory, when executed individually or collectively by the at least one processor, causes the electronic device to: In response to receiving a search assistant activation input while outputting the first screen through the above display, activate the search assistant, While the search assistant is activated, in response to receiving a first user input for a first area of ​​the first screen, identify a feature of a search target corresponding to the first user input, According to the characteristics of the above-identified search target, a first search condition for the first area of ​​the first screen is obtained and added to the search box, A pop-up window displaying the search box and search conditions included in the search box is displayed by overlapping a portion of the first screen, Through the above communication circuit, the search conditions included in the search box are transmitted as a query to the search assistant server, and An electronic device that stores instructions that cause search results received from the search assistant server to be displayed overlapping another part of the first screen through the communication circuit.

14. In paragraph 13, The memory, when individually or collectively executed by the at least one processor, causes the electronic device to: An electronic device storing commands that cause the first user input to include at least one of a touch input, a voice input, a text input, a motion input, a keyboard input, a mouse input, a stylus pen input, a biosignal input, or a wearable device input.

15. In paragraph 13, The memory, when individually or collectively executed by the at least one processor, causes the electronic device to: An electronic device in which the search box and the pop-up window displaying the search conditions included in the search box store commands that cause a thumbnail, attribute information, and object information to be displayed for each of the search conditions.

Images