Electronic device, method, and recording medium for providing emotion information on basis of content

Abstract

According to an embodiment of the present disclosure, provided are an electronic device, method, and recording medium for searching for content (e.g.: an image and / or video) on the basis of an emotion tag, and providing emotion information related to the content. The electronic device, in response to an input of a search word requesting to search for content related to emotion, extracts an emotion keyword from the search word. The electronic device searches for the content in the electronic device on the basis of the emotion keyword, and groups the content on the basis of time and place. The electronic device generates, on the basis of the content for each group, a prompt for inferring a cause of an emotion related to the emotion keyword, and provides the content for each group and the prompt to a generative AI. The electronic device obtains data on the basis of the content for each group and the prompt. The electronic device displays, on a display on the basis of the data, an interface including the content for each group and the data.

Description

Electronic device, method, and recording medium for providing emotional information based on content

[0001] Embodiments of the present disclosure provide an electronic device, method, and recording medium for searching content (e.g., images and / or videos) based on an emotion tag and providing emotional information related to the content.

[0002] With the advancement of digital technology, various types of electronic devices, such as smartphones, tablet PCs (personal computers), laptop computers, desktop computers, digital cameras, and / or wearable devices, are becoming widely used. The hardware and / or software components of these electronic devices are continuously being developed to support and enhance their functionality.

[0003] For example, portable electronic devices (hereinafter referred to as "electronic devices"), such as smartphones, can now be equipped with a variety of functions. Electronic devices include touchscreen-based displays that allow users to easily access various functions, and can display screens for various applications through these displays.

[0004] Electronic devices can display content (e.g., images and / or videos) via a display. In one embodiment, the content may include tags, and the electronic device can use the tags to search for and provide content containing the same tags within the electronic device. For example, the electronic device can search for only content containing tags identical to those contained in the content and provide the searched content to the user by simply displaying the searched content on the display.

[0005] The above information may be provided as background art to aid in understanding the present disclosure. No claim or determination is made as to whether any of the above is applicable as prior art related to the present disclosure.

[0006] In one embodiment of the present disclosure, an electronic device for providing emotional information based on content (e.g., images and / or videos), a method of operating the same, and a recording medium are provided.

[0007] In one embodiment of the present disclosure, an electronic device, an operating method thereof, and a recording medium are provided for searching content based on an emotion tag and providing emotional information regarding the cause of the emotion of the content.

[0008] In one embodiment of the present disclosure, an electronic device, a method of operation thereof, and a recording medium are provided for searching and providing content related to a requested emotion by using tags related to the content (e.g., user-defined tags and / or hidden tags based on analysis of the electronic device) in response to a request for content search based on emotion.

[0009] In one embodiment of the present disclosure, an electronic device, a method of operation thereof, and a recording medium are provided, which sort and group emotion-based contents based on time and place, and generate and provide emotion information (e.g., a description) about the cause of the emotion (e.g., how it is related to a requested emotion) based on the contents by group.

[0010] The technical problems to be achieved in this document are not limited to the technical problems mentioned above, and other technical problems not mentioned can be clearly understood by a person having ordinary skill in the technical field to which the present invention belongs from the description below.

[0011] An electronic device according to an embodiment of the present disclosure may include a display, at least one processor including processing circuitry, and a memory storing instructions. In one embodiment, the memory may store instructions that, when individually and / or collectively executed by the at least one processor, cause the electronic device to perform operations.

[0012] According to one embodiment, the instructions, when executed by the at least one processor, may cause the electronic device to receive an input of a search term requesting a search for content related to an emotion. According to one embodiment, the instructions, when executed by the at least one processor, may cause the electronic device, in response to the input, to extract an emotion keyword from the search term. According to one embodiment, the instructions, when executed by the at least one processor, may cause the electronic device to perform a content search within the electronic device based on the emotion keyword. According to one embodiment, the instructions, when executed by the at least one processor, may cause the electronic device to group searched content based on time and location. According to one embodiment, the instructions, when executed by the at least one processor, may cause the electronic device to generate a prompt for inferring a cause of an emotion related to the emotion keyword based on content for each group. According to one embodiment, the instructions, when executed by the at least one processor, may cause the electronic device to transmit the group-specific content and the prompt to an on-device and / or server-generated artificial intelligence (AI). According to one embodiment, the instructions, when executed by the at least one processor, may cause the electronic device to obtain data based on the group-specific content and the prompt. According to one embodiment, the instructions, when executed by the at least one processor, may cause the electronic device to display an interface including the group-specific content and data on a display based on the data.

[0013] An operating method of an electronic device according to an embodiment of the present disclosure may include receiving an input of a search word requesting a content search related to an emotion. The operating method may include extracting an emotional keyword from the search word in response to the input. The operating method may include performing a content search within the electronic device based on the emotional keyword. The operating method may include grouping the searched content based on time and location. The operating method may include generating a prompt for inferring the cause of the emotion related to the emotional keyword based on the content for each group. The operating method may include transmitting the content for each group and the prompt to a generative artificial intelligence (AI) of an on-device and / or a server. The operating method may include acquiring data based on the content for each group and the prompt. The operating method may include displaying an interface including the content for each group and the data on a display based on the data.

[0014] In order to solve the above-described problem, various embodiments of the present disclosure may include a computer-readable recording medium having recorded thereon a program for executing the method on a processor.

[0015] According to one embodiment, a non-transitory computer-readable recording medium (or storage medium or computer program product) storing one or more programs is described. According to one embodiment, the one or more programs may include instructions for performing the following actions: receiving an input of a search word requesting a search for content related to an emotion; extracting an emotion keyword from the search word in response to the input; performing a content search within an electronic device based on the emotion keyword; grouping the searched content based on time and location; generating a prompt for inferring a cause of an emotion related to the emotion keyword based on content by group; transmitting the content by group and the prompt to a generative artificial intelligence (AI) of an on-device and / or a server; acquiring data based on the content by group and the prompt; and displaying an interface including the content by group and the data on a display based on the data.

[0016] Further scope of the applicability of the present disclosure will become apparent from the detailed description below. However, since various modifications and variations within the spirit and scope of the present disclosure will readily become apparent to those skilled in the art, it should be understood that the detailed description and specific examples, such as preferred embodiments of the present disclosure, are given by way of example only.

[0017] According to an embodiment of the present disclosure, an electronic device, an operation method thereof, and a recording medium thereof, the electronic device may support searching for content that includes tags identical to and / or similar to tags related to a search keyword. According to an embodiment, the electronic device may, in response to a request from a user to search for content (e.g., images and / or videos) based on emotions, search for content using emotion tags written (or generated) at various points in time. According to an embodiment, the electronic device may group searched images by time and place, additionally acquire more meaningful content, and generate (or acquire) an explanation of the cause of emotion (or emotion information) using an AI model (e.g., generative AI) and provide the same to the user. According to an embodiment, the electronic device may perform an operation of summarizing the explanation of the cause of emotion for each entire group into one and provide the summarized explanation to the user. According to an embodiment, the electronic device may not only search for and provide content, but may also organize and sort meaningful content and information and provide interpretations of emotions to the user. According to one embodiment, a new UX (user experience) / UI (user interface) can be provided to users by providing emotional cause descriptions using emotion-based content search and generative AI.

[0018] In addition, various effects may be directly or indirectly realized through this document. The effects obtained through this disclosure are not limited to those mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art to which this disclosure pertains, based on the description below.

[0019] In connection with the description of the drawings, the same or similar reference numerals may be used for the same or similar components.

[0020] FIG. 1 is a block diagram of an electronic device within a network environment according to various embodiments.

[0021] FIG. 2 is a block diagram illustrating an integrated intelligence system according to one embodiment.

[0022] FIG. 3 is a block diagram illustrating a generative artificial intelligence system according to one embodiment.

[0023] FIG. 4 is a diagram schematically illustrating the configuration of an electronic device according to one embodiment of the present disclosure.

[0024] FIG. 5 is a diagram illustrating an example of providing emotional information using generative AI in an electronic device according to one embodiment of the present disclosure.

[0025] FIG. 6 is a flowchart illustrating a method of operating an electronic device according to one embodiment of the present disclosure.

[0026] FIG. 7 is a flowchart illustrating a method of operating an electronic device according to one embodiment of the present disclosure.

[0027] FIG. 8 is a flowchart illustrating a method of operating an electronic device according to one embodiment of the present disclosure.

[0028] FIG. 9 is a diagram illustrating an example of an operation of searching for and providing emotion-based content in an electronic device according to one embodiment of the present disclosure.

[0029] FIG. 10 is a diagram illustrating an example of an operation of searching for and providing emotion-based content in an electronic device according to one embodiment of the present disclosure.

[0030] FIG. 11 is a diagram illustrating an example of an operation of generating and providing emotional information related to search content in an electronic device according to one embodiment of the present disclosure.

[0031] FIG. 12 is a diagram illustrating an example of an operation of providing content group-specific descriptions in an electronic device according to one embodiment of the present disclosure.

[0032] FIG. 13 is a diagram illustrating an example of an interface provided by an electronic device according to one embodiment of the present disclosure.

[0033] FIG. 14 is a diagram illustrating an example of an interface provided by an electronic device according to one embodiment of the present disclosure.

[0034] FIG. 15 is a flowchart illustrating an operation method of an electronic device according to one embodiment of the present disclosure.

[0035] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. In connection with the description of the drawings, the same or similar reference numerals may be used for identical or similar components. Furthermore, in the drawings and related descriptions, descriptions of well-known functions and configurations may be omitted for clarity and conciseness.

[0036] FIG. 1 is a block diagram of an electronic device (101) within a network environment (100) according to various embodiments.

[0037] Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of an electronic device (104) or a server (108) via a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0038] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or operations. According to one embodiment, as at least a part of the data processing or operations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit (CPU) or an application processor (AP)) or an auxiliary processor (123) (e.g., a graphic processing unit (GPU), a neural processing unit (NPU), an image signal processor (ISP), a sensor hub processor, or a communication processor (CP)) that can operate independently or together with the main processor (121). For example, when the electronic device (101) includes the main processor (121) and the auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a given function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0039] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, the artificial intelligence model may include a software structure.

[0040] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0041] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (OS) (142), middleware (144), or an application (146).

[0042] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0043] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0044] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. In one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0045] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0046] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0047] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, a secure digital (SD) card interface, or an audio interface.

[0048] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0049] A haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. In one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0050] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0051] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least a part of a power management integrated circuit (PMIC).

[0052] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0053] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or a wide area network (WAN))). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0054] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). NR access technology can support high-speed transmission of high-capacity data (eMBB, enhanced mobile broadband), minimizing terminal power and connecting multiple terminals (mMTC, massive machine type communications), or high reliability and low latency communications (URLLC, ultra-reliable and low-latency communications). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0055] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas by, for example, the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0056] According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high frequency band.

[0057] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0058] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server using machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0059] FIG. 2 is a block diagram illustrating an integrated intelligence system according to one embodiment.

[0060] Referring to FIG. 2, an integrated intelligent system of one embodiment may include an electronic device (201) (e.g., the electronic device (101) of FIG. 1), an intelligent server (300), and a service server (399).

[0061] According to the illustrated embodiment, the electronic device (201) may include a communication interface (210), an input / output (I / O) interface (220), a processor (230), and / or a memory (240). The components listed above may be operatively or electrically connected to each other. For example, the electronic device (201) may include at least some of the components of the electronic device (101) of FIG. 1.

[0062] The communication interface (210) can be connected to an external device (e.g., an intelligent server (300) and / or a service server (399)) via a network (299) (e.g., any network including a cellular network and / or a wireless local area network (WLAN)) to transmit and receive data. For example, the communication interface (210) can correspond to the CP and / or communication circuit of FIG. 1. The I / O interface (220) can receive user input, process received user input, and / or output a result processed by the processor (230) using an input / output device (not shown) (e.g., a microphone, a speaker, and / or a display (e.g., a display of FIG. 1).

[0063] The processor (230) may be operatively or electrically connected to a communication interface (210), an I / O interface (220), and / or a memory (240) (e.g., the memory (130) of FIG. 1) to perform a designated operation. For example, the processor (230) may correspond to the processor (120) of FIG. 1. The processor (230) may execute a program (or one or more instructions) stored in the memory (240) to perform a designated operation. For example, the processor (230) may receive a user's voice input (e.g., a user's speech) through the I / O interface (220) or from an external electronic device. The processor (230) may transmit the voice input received through the communication interface (210) to the intelligent server (300). For example, the processor (230) may include one or more processors.

[0064] The processor (230) may receive a result corresponding to the voice input from the intelligent server (300). For example, the processor (230) may receive a plan corresponding to the voice input and / or a result calculated using the plan from the intelligent server (300). For example, the plan may include, but is not limited to, information regarding a plurality of sequential operations to be executed by the first electronic device (201) and / or another electronic device in relation to the voice input. The processor (230) may receive a request from the intelligent server (300) to obtain information (e.g., entities, slots, and / or parameters) necessary to generate a plan corresponding to the voice input. The processor (230) may transmit the necessary information to the intelligent server (300) in response to the request.

[0065] The processor (230) may visually, tactilely, and / or audibly output the results of executing the operations specified according to the plan via the I / O interface (220). For example, the processor (230) may sequentially display the execution results of multiple operations on the display. As an example, the processor (230) may display only the execution results of executing multiple operations (e.g., the execution results of one of the multiple operations or the last operation) on the display.

[0066] The processor (230) can recognize voice input. For example, the processor (230) can execute an intelligent app (or a voice recognition app) to process the voice input in response to a specified voice input (e.g., "Wake up!"). The processor (230) can provide a voice recognition service through the intelligent app. The processor (230) can transmit the voice input to the intelligent server (300) through the intelligent app and receive a result corresponding to the voice input from the intelligent server (300).

[0067] An intelligent server (300) of one embodiment can receive a user's voice input from an electronic device (201) via a network (299). The intelligent server (300) can convert audio data corresponding to the received voice input into text data. The intelligent server (300) can generate at least one plan for performing a task corresponding to the user's voice input based on the text data. The intelligent server (300) can transmit the generated plan or a result according to the generated plan to the electronic device (201) via the network (299).

[0068] An intelligent server (300) of one embodiment may include a front end (310), a natural language platform (320), a capsule database (330), an execution engine (340), and / or an end user interface (350).

[0069] The front end (310) can receive a voice input received by the electronic device (201) from the electronic device (201). The front end (310) can transmit a response corresponding to the voice input to the electronic device (201).

[0070] The natural language platform (320) may include an automatic speech recognition (ASR) module (321), a natural language understanding (NLU) module (323), a planner module (325), a natural language generator (NLG) module (327), and / or a text-to-speech (TTS) module (329).

[0071] The automatic speech recognition module (321) can convert the voice input received from the electronic device (201) into text data. The natural language understanding module (323) can identify the user's intent and / or parameters (e.g., entities and / or slots) based on the text data of the voice input. The user's intent corresponds to the voice input and may include information indicating an action (or function) that the user wishes to perform using the device. The slot may be detailed information related to the user's intent. The slot may be acquired based on a domain corresponding to the utterance. The slot may be variable information required to perform the action. In one embodiment, the variable information constituting the slot may include a named entity.

[0072] The planner module (325) can generate a plan using the intent and / or parameters determined by the natural language understanding module (323). For example, the planner module (325) can determine at least one domain necessary to perform a task based on the determined intent. The planner module (325) can determine a plurality of operations included in each of the at least one domain determined based on the intent. The domain may correspond to a category (or service) associated with an operation (or function) that the user wishes to perform using the device. The domain may be classified according to a service (e.g., an app) related to the text. The domain may be related to the user's intent corresponding to the text. The domain may be classified according to, for example, the type of application that received the voice input and / or the type of service to be provided based on the voice input, but is not limited thereto. In one example, the determination of the domain may be performed by another module (e.g., the natural language understanding module (323)). The planner module (325) can determine parameters required to execute a plurality of determined actions or result values ​​output by the execution of the plurality of actions. The parameters and result values ​​can be defined as concepts of a specified format (or class). For example, the plan can include a plurality of actions and / or a plurality of concepts determined by the user's intention. The planner module (325) can determine the relationship between the plurality of actions and / or the plurality of concepts in a step-by-step (or hierarchical) manner. For example, the planner module (325) can identify the execution order of the plurality of actions (e.g., the plurality of actions determined based on the user's intention) based on the plurality of concepts (e.g., parameters required to execute the plurality of actions and results output by the execution of the plurality of actions). The planner module (325) can generate a plan including association information (e.g., ontology) between the plurality of actions and the plurality of concepts.The planner module (325) can create a plan using information (e.g., at least one capsule) stored in a capsule database (330) in which a set of relationships between concepts and actions is stored.

[0073] The planner module (325) can generate a plan based on an artificial intelligence (AI) system. For example, the AI ​​system can include one or more electronic devices and / or one or more processing circuits to execute a rule-based system, a neural network-based system (e.g., a feedforward neural network (FNN), a recurrent neural network (RNN)), or a combination thereof. The AI ​​system described above is exemplary, and the AI ​​system can be an AI system based on any machine learning-based model. The planner module (325) can select a plan corresponding to a user request from a set of predefined plans, or generate a plan in real time in response to a user request.

[0074] The natural language generation module (327) can convert specified information into text format. The information converted into text format may be in the form of natural language speech. The text-to-speech conversion module (329) can convert information in text format into information in speech format.

[0075] The capsule database (330) can store information on the relationship between multiple concepts and actions corresponding to multiple domains (e.g., applications). The capsule database (330) can store at least one capsule (e.g., capsule (331) and / or capsule (333)) in the form of a concept action network (CAN). For example, the capsule database (330) can store actions for processing tasks corresponding to a user's voice input and parameters required for the actions in the form of a CAN. A capsule can include multiple action objects (or action information) and / or concept objects (or concept information) included in a plan. For example, capsules (331, 333) can be created for each domain and stored in the capsule database (330), but are not limited thereto.

[0076] The execution engine (340) can produce results using the generated plan. The end user interface (350) can transmit the produced results to the electronic device (201).

[0077] According to one embodiment, some functions (e.g., the natural language platform (320)) or all functions of the intelligent server (300) may be implemented in the electronic device (201). For example, the electronic device (201) may execute one or more programs including the natural language platform (e.g., the natural language platform (320) of FIG. 2) separately from the intelligent server (300). For example, the electronic device (201) may directly perform at least some of the operations of the natural language platform (320) of the intelligent server (300) (e.g., the automatic speech recognition module (321), the natural language understanding module (323), the planner module (325), the natural language generation module (327), and / or the text-to-speech module (329)).

[0078] In one embodiment, a service server (399) may provide a service (e.g., food ordering or hotel reservation) designated to an electronic device (201). The service server (399) may be a server operated by a different operator than the intelligent server (300). The service server (399) may communicate with the intelligent server (300) and / or the electronic device (201) via a network (299). The service server (399) may communicate with the intelligent server (300) via a separate connection (not shown). The service server (399) may provide the intelligent server (300) with information for generating a plan corresponding to a voice input received in the electronic device (201) (e.g., operation information and / or concept information for providing a designated service). The provided information may be stored in a capsule database (330). The service server (399) may provide the intelligent server (300) with result information according to the plan received from the electronic device (201).

[0079] FIG. 3 is a block diagram illustrating a generative artificial intelligence system according to one embodiment.

[0080] Referring to FIG. 3, a generative artificial intelligence system (e.g., an intelligent server, server (108) of FIG. 1) according to one embodiment may include a user interface (260), a database (265), an applications / service component (270), an AI framework (280), and a generative AI model (290).

[0081] The user interface (260) can receive a user query. The user query can be in the form of natural language, images, and videos. Additionally, context information can be transmitted along with the user query. As another example, the user query can also be a non-natural language input that does not generate natural language, such as a design request or modification. Furthermore, the user interface (260) can also be in the form of a mixture of natural language, images, sounds, and context information as described above. Furthermore, the user interface (260) can output the results of the generative artificial intelligence system to the user. The output can be in the form of natural language or specific content, and can also be provided in the form of an action requested by the user.

[0082] The AI ​​framework (280) can receive a user query and coordinate and control each component necessary to carry out the user's intent. The AI ​​framework (280) may include a prompt design component (281), an application and plug-in management component (APIs / Plugins Management component) (283), and an output modification component (285).

[0083] A user query or action entered in the user interface (260) can be transmitted to a prompt design component (281). The prompt design component (281) can be used to generate prompts suitable for input into a large language model (LLM) or a large multimodal model (LMM). The prompt design component (281) can be an AI component that uses a machine learning algorithm or a neural network to develop better prompts over time. The prompt design component (281) can access a knowledge component containing user preference data, a prompt library, and prompt examples to generate prompts and transmit them to the large language model (LLM) or the large multimodal model (LMM).

[0084] The application and plugin management component (283) can communicate with external information when a request for additional information is made when user input is passed as input to the generative model. The application and plugin management component (283) establishes a channel for communication with the AI ​​Interface externally via an application programming interface (API), thereby enabling access to various data sources. Furthermore, the application and plugin management component (283) can request an action via the API to ultimately perform a user query, rather than an intermediate result, if the application or service needs to perform that action. Information obtained from external sources can be passed as input to the generative model along with user input.

[0085] The output modification component (285) can fine-tune the output from the generative model. For example, the output modification component (285) can verify that the content generated through a language model (LLM) or a large-scale multi-modal model (LMM) is not irrelevant, does not contain biased content, or does not contain harmful content. In addition, the output modification component (285) can determine to what extent the content matches the user's desired result and can proceed with additional processing if necessary. Additionally, the output modification component (285) can configure and provide the user with hints to avoid undesired output.

[0086] A generative AI model (290) generally refers to an artificial intelligence neural network that creates new types of data based on user input information. Representative models that generate images include the generative adversarial network (GAN) and the variational auto encoder (VAE). Recently, diffusion-based generative models that use VAE and Transformer structures are called generative models. In addition, language models are models trained to statistically output the most appropriate output based on input values, and representative examples include models such as CHAT-GPT 3 and CHAT-GPT 4. In addition, since they can recognize various types of data input, such as text, images, and voice, and generate new data corresponding to them, they are called LMMs (large multimodal models).

[0087] FIG. 4 is a diagram schematically illustrating the configuration of an electronic device according to one embodiment of the present disclosure.

[0088] According to one embodiment, FIG. 4 may represent a block diagram of an exemplary electronic device (400) (e.g., electronic device (101) of FIG. 1) capable of performing the operations described herein.

[0089] Referring to FIG. 4, the electronic device (400) may be one of various forms of electronic devices, such as a notebook (490), smartphones (491) having various form factors (e.g., a bar-type smartphone (491-1), a foldable-type smartphone (491-2), or a sliderable (or rollable) type smartphone (491-3)), a tablet (492), a cellular phone (not shown), and other similar computing devices (not shown). The components, their relationships, and their functions illustrated in FIG. 4 are exemplary only and do not limit the implementations described or claimed in this document. The electronic device (400) may be referred to as a mobile device, a user device, a multi-function device, a portable device, or a server.

[0090] According to one embodiment, the electronic device (400) may include all or at least a portion of the components of the electronic device (400) as described in the description with reference to FIG. 1. For example, in various embodiments of the present document, some of the illustrated components may be omitted or replaced. The electronic device (400) may include at least a portion of the components and / or functions of the electronic device (101) of FIG. 1. At least some of the individual components of the illustrated (or not illustrated) electronic device (400) may be operatively, functionally, and / or electrically connected to each other.

[0091] The electronic device (400) may include components including at least one processor (410) (e.g., processor (120) of FIG. 1) (hereinafter, referred to as processor (410)), at least one memory (420) (e.g., memory (130) of FIG. 1) (hereinafter, referred to as memory (420)), at least one display (440) (e.g., display module (160) of FIG. 1) (hereinafter, referred to as display (440)), at least one image sensor (450) (hereinafter, referred to as image sensor (450)), at least one communication circuit (460) (e.g., communication module (190) of FIG. 1) (hereinafter, referred to as communication circuit (460)), and / or at least one sensor (470) (hereinafter, referred to as sensor (470)). The above components are merely exemplary. For example, the electronic device (400) may include other components (e.g., power management integrated circuitry (PMIC), audio processing circuitry, an antenna, a rechargeable battery, or input / output interfaces). For example, some components may be omitted from the electronic device (400). For example, some components may be integrated into a single component.

[0092] The processor (410) may perform application layer processing functions requested by a user of the electronic device (400). According to one embodiment, the processor (410) may provide control and commands of functions for various blocks of the electronic device (400). According to one embodiment, the processor (410) may perform operations or data processing related to control and / or communication of each component of the electronic device (400). For example, the processor (410) may include at least some of the configurations and / or functions of the processor (120) of FIG. 1. According to one embodiment, the processor (410) may be operatively connected to components of the electronic device (400). According to one embodiment, the processor (410) may load commands or data received from other components of the electronic device (400) into the memory (420), process the commands or data stored in the memory (420), and store result data.

[0093] The processor (410) may be implemented as one or more integrated circuit (or circuitry) chips and may perform various data processing operations. The processor (410) may include at least one electrical circuit and may individually and / or collectively perform distributed processing of instructions (or programs, data, etc.) stored in the memory (420). The processor (410) may include a processor assembly including one or more processing circuitries and / or executable program elements.

[0094] The processor (410) may include any processing circuit operative to control the performance and operations of one or more components (e.g., memory (420), display (440), image sensor (450), communication circuit (460), and / or sensor (470)) of the electronic device (400). For example, the processor (410) may be an application processor (AP). For example, the processor (410) may be a system semiconductor that is responsible for various functions (e.g., computation and multimedia driving functions) of the electronic device (400). The processor (410) may be implemented as a system on chip (SoC) (e.g., one chip or chipset). For example, the processor (410) may be implemented as a plurality of cores (or at least one core circuit), a plurality of chips, or a plurality of chipsets. For example, the processor (410) may include one or more processing circuits. For example, the processor (410) may include one or more processing circuits configured to individually and / or collectively perform various functions of the present disclosure. As a non-limiting example, at least a portion of the processor (410) may be included in a first chip of the electronic device (400), and at least another portion of the processor (410) may be included in a second chip of the electronic device (400), different from the first chip of the electronic device (400).

[0095] For example, the processor (410) may include a central processing unit (CPU) (411), a graphics processing unit (GPU) (412), a neural processing unit (NPU) (413), an image signal processor (ISP) (414), a display controller (415), a memory controller (416), a storage controller (417), a communication processor (CP) (418), and / or a sensor interface (419). These components of the processor (410) are merely exemplary. For example, the processor (410) may further include other components. For example, some components of the processor (410) may be omitted from the processor (410). For example, some components of the processor (410) may be included as separate components of the electronic device (400) outside the processor (410). For example, some components of the processor (410) (e.g., memory controller (416)) may be included within other components (e.g., at least a portion of memory (420), an interface (e.g., available for connection to at least one component of the electronic device (400)), a display (440) and / or an image sensor (450)).

[0096] The processor (410) can cause other components of the electronic device (400) to perform various operations by executing instructions stored in the memory (420).

[0097] The CPU (411) (or central processing circuit) may be configured to control components of the processor (410) based on the execution of instructions stored in the memory (420) (e.g., volatile memory (421) and / or non-volatile memory (422)). The CPU (411) may decipher user commands and perform arithmetic and logical operations, and / or data processing operations. For example, the CPU (411) may be responsible for functions such as memory, interpretation, operation, and control. The CPU (411) may execute all software (e.g., application (146) of FIG. 1) of the electronic device (400) on top of an operating system (OS) and control hardware devices.

[0098] The CPU (411) may store commands or data in a volatile memory (421) of the memory (420) (e.g., the volatile memory (132) of FIG. 1) as at least part of data processing or calculation, process the commands or data stored in the volatile memory (421), and store result data in a non-volatile memory (422) of the memory (420) (e.g., the non-volatile memory (134) of FIG. 1).

[0099] The CPU (411) may include a single processor core or multiple processor cores (multi-core). The CPU (411) may be a programmable processor that stores executable instructions (e.g., instructions capable of performing operations of the CPU (411)) and executes the instructions.

[0100] The CPU (411) can operate in a multi-domain environment. The CPU (411) can operate in a multi-domain environment of a normal world (e.g., a non-secure world, a framework, or a non-secure environment) and a secure world (e.g., a secure framework or a secure environment). In one embodiment, a domain of the secure world can include one or more domains (e.g., a trusted OS, a trust zone, and / or a virtualization framework).

[0101] The GPU (412) (or graphics processing circuit) may be configured to execute parallel operations (e.g., rendering). The GPU (412) may be responsible for graphics processing. The GPU (412) may receive commands from the CPU (411) and perform graphics processing to express the shape, position, color, shading, movement, and / or texture of objects (or objects) on the display (440).

[0102] The NPU (413) (or neural processing circuit, or AI (artificial intelligence) chip) may be configured to execute operations (e.g., convolution computation) for an artificial intelligence model. The NPU (413) may perform processing optimized for an artificial intelligence deep-learning algorithm. The NPU (413) is a processor optimized for deep-learning algorithm operations (e.g., artificial intelligence operations) and can process big data quickly and efficiently like a human neural network. For example, the NPU (413) may be mainly used for artificial intelligence operations. The NPU (413) may recognize objects, environments, and / or people in the background when taking a video through a camera and automatically adjust the focus, automatically switch the camera's shooting mode to food mode when taking a picture of food, and / or perform processing to erase only unnecessary subjects from the captured results. The NPU (413) may perform processing to generate an answer based on given information (e.g., natural language).

[0103] The ISP (414) (or image signal processing circuit) may be configured to process a raw image acquired through the image sensor (450) into a format suitable for a component within the electronic device (400) or a component of the processor (410). For example, the ISP (414) may be responsible for image processing and correction of images and videos. The ISP (414) may correct unprocessed data (e.g., raw data) transmitted from the image sensor (450) of a camera (e.g., the camera module (180) of FIG. 1) to generate an image in a form more preferred by the user. The ISP (414) may perform post-processing, such as adjusting partial brightness of the image and emphasizing detailed parts. For example, the ISP (414) may independently perform a process of tuning and correcting the image quality of an image acquired through the camera to generate a result preferred by the user.

[0104] The ISP (414) may support artificial intelligence-based image processing technology. The ISP (414) may support scene segmentation (e.g., image segmentation) technology that recognizes and / or classifies parts of a scene being captured in conjunction with the NPU (413). For example, the ISP (414) may include a function for processing objects such as the sky, bushes, and / or skin by applying different parameters to them. The ISP (414) may detect and display a human face during image capture using the artificial intelligence function, or adjust the brightness, focus, and / or color of the image using the coordinates and information of the face.

[0105] According to one embodiment, the electronic device (400) can support integrated machine learning processing by interacting with all processors such as the CPU (411), GPU (412), NPU (413), and ISP (414).

[0106] The display controller (415) (or display control circuit, or DPU (display processing unit)) may be configured to process an image obtained from the CPU (411), GPU (412), ISP (414), or memory (420) (e.g., volatile memory (421)) into a format suitable for the display (440).

[0107] The memory controller (416) (or memory control circuit) may be configured to control reading data from the volatile memory (421) and writing data to the volatile memory (421).

[0108] The storage controller (417) (or storage control circuit) may be configured to control reading data from the nonvolatile memory (422) and writing data to the nonvolatile memory (422).

[0109] The CP (418) (or communication processing circuit) may be configured to process data obtained from a component of the processor (410) into a format suitable for transmission to another electronic device via the communication circuit (460), or to process data obtained from another electronic device via the communication circuit (460) into a format suitable for processing by the component of the processor (410). For example, the communication circuit (460) may include one or more communication circuits.

[0110] The sensor interface (419) (or sensing data processing circuit, sensor hub) may be configured to process data about the state of the electronic device (400) and / or the state of the surroundings of the electronic device (400), obtained through the sensor (470), into a format suitable for components of the processor (410).

[0111] According to one embodiment, the processor (410) is operable in a normal mode (or normal world) and a secure mode (or secure world). According to one embodiment, the processor (410) may control (or process) the overall operations related to searching for content (e.g., images and / or videos) based on emotion tags and providing emotion cause descriptions (e.g., emotion information) using generative artificial intelligence (AI), based on processing circuits and / or executable program elements.

[0112] According to one embodiment, the processor (410) may receive an input of a search term requesting a search for content related to an emotion. According to one embodiment, the processor (410) may extract an emotion keyword from the search term in response to the input of the search term. According to one embodiment, the processor (410) may search for content within the electronic device (400) based on the emotion keyword and group the content based on time and location. According to one embodiment, the processor (410) may generate a prompt (e.g., a text prompt) for inferring the cause of the emotion related to the emotion keyword based on the content for each group, and may provide (or transmit) the content for each group and the prompt to a generative AI of an on-device and / or a server. According to one embodiment, the processor (410) may obtain data (or result data) (e.g., a description of the cause of the emotion) based on the content for each group and the prompt. According to one embodiment, the processor (410) may display an interface including group-specific content and data on the display (440) based on the data.

[0113] According to one embodiment, the processor (410) may generate a prompt based on whether the searched content has a defined condition. According to one embodiment, the processor (410) may determine whether the searched content satisfies the defined condition. According to one embodiment, if the defined condition is not satisfied, the processor (410) may additionally search for similar content by group based on time and / or location related to the content within the group that does not satisfy the defined condition. For example, if the processor (410) determines that the number of contents within each group is less than a predetermined number and / or that it is impossible to generate an emotion-based reason explanation (or emotion information) using an AI model (e.g., generative AI) using the content, the processor (410) may additionally search for and acquire similar content based on time and / or location within each group.

[0114] According to one embodiment, the detailed operation of a processor (410) (e.g., processor (120) of FIG. 1) of an electronic device (400) (e.g., electronic device (101) of FIG. 1) is described with reference to the drawings described below.

[0115] According to one embodiment, the operations performed by the processor (410) may be implemented by executing instructions stored in a recording medium (or a computer program product or storage medium). For example, the recording medium may include a non-transitory computer-readable recording medium having recorded thereon a program for executing various operations performed by the processor (410).

[0116] The embodiments described in the present disclosure may be implemented in a computer-readable recording medium using software, hardware, or a combination thereof. In a hardware implementation, the operations described in one embodiment may be implemented using at least one of application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, micro-controllers, microprocessors, and / or other electrical units for performing functions.

[0117] In one embodiment, a computer-readable recording medium (or computer program product or storage medium) is provided, which records a program for causing an electronic device (400) to perform (or execute) various operations.

[0118] The above operations may include: receiving an input of a search term requesting a search for content related to an emotion (e.g., images and / or videos); extracting an emotion keyword from the search term in response to the input; performing a content search within the electronic device (400) based on the emotion keyword; grouping the searched content based on time and place; generating a prompt (e.g., a text prompt) for inferring a cause of an emotion related to the emotion keyword based on the content by group; transmitting the content by group and the prompt to a generative AI of an on-device and / or a server; acquiring data based on the content by group and the prompt; and displaying an interface including the content by group and the data on a display (440) based on the data.

[0119] The memory (420) includes at least a portion of the configuration and / or function of the memory (130) of FIG. 1, and may store software (e.g., the program (140) of FIG. 1 and / or the application (146) of FIG. 1). The memory (420) may include one or more storage media (or one or more storage devices). For example, the memory (420) may include a memory assembly including one or more storage media. For example, the one or more storage media may include permanent memory (e.g., non-volatile memory (422)) such as a hard drive, flash memory, read-only memory (ROM), semi-permanent memory (e.g., volatile memory (421)) such as random access memory (RAM), any other suitable type of storage (or storage assembly), or any combination thereof.

[0120] The memory (420) may include a cache memory, which is one or more different types of memory used to temporarily store data for a function or feature of the electronic device (400). As a non-limiting example, the cache memory may be included within the processor (410).

[0121] The memory (420) may be fixedly embedded within the electronic device (400) or incorporated into one or more suitable types of components (e.g., a subscriber identity module (SIM) card and / or a secure digital (SD) card) that can be repeatedly inserted into and removed from the electronic device (400).

[0122] For example, the memory (420) may store one or more software applications, such as an operating system (OS) (or system) software application, a firmware software application, a driver software application, a plug-in (e.g., add-in, add-on, and / or applet) software application, and / or any other suitable software applications. For example, the one or more software applications may include instructions executable by the processor (410). For example, the memory (420) may store instructions callable by an application programming interface (API). For example, the memory (420) may store instructions within a library.

[0123] The memory (420) can store various data used by at least one component (e.g., processor (410)) of the electronic device (400). In one embodiment, the data can include input data or output data for software (e.g., program (140) of FIG. 1 (e.g., operating system (142), middleware (144), and / or application (146) of FIG. 1)) and commands related to the software.

[0124] The memory (420) may include a volatile memory (421) (e.g., the volatile memory (132) of FIG. 1) or a non-volatile memory (422) (e.g., the non-volatile memory (134) of FIG. 1). The memory (420) may store commands or data received from the processor (410) in the volatile memory (421), and may store result data processed by the processor (410) of the commands or data stored in the volatile memory (421) in the non-volatile memory (422).

[0125] According to one embodiment, the memory (420) may include at least one storage, including a first storage portion corresponding to a normal mode and a second storage portion corresponding to a secure mode. The first storage portion and the second storage portion may be different areas of a single storage, or may be separate storages. According to one embodiment, data generated in the normal mode may be stored in the first storage portion. According to one embodiment, data stored in the first storage portion may be stored in the second storage portion in the secure mode.

[0126] In one embodiment, the data may include content including images and / or videos, tag information regarding tags set for each piece of content (e.g., sentiment tags), sentiment data, and group information regarding grouping of content based on time and / or location.

[0127] In one embodiment, emotional data may be defined as shown in the example in Table 1 below. For example, emotional data may be categorized by emotion type (e.g., a representative emotion), similar emotions related to the representative emotion, or emotional terms. For example, representative emotions may be categorized as joy, sadness, surprise, fear, disgust, and anger, and similar emotions may include one or more similar emotions for each representative emotion.

[0128] Representative emotionsSimilar emotions (or emotion terms)JoyJoyful, laughter (laughter), pleasure (pleasant), smile (smile), pleased (pleased), excited, affectionate, glad, like (good), happiness (happy), love (love, excitement (excited), sadnessSad, sorrowful (sad), sorrowful (sad), crying (cry), sorrowful (sorrowful), tears (tears), upset (upset), frustration (frustration), pain (pain), anxiety (anxious), disappointment (disappointed), worry (disappointed), loneliness (lonely), fatigue (tired / difficult), tired (tired / weary), surpriseSurprised, embarrassed (embarrassed), dizzyFearFearful (scary), fear (fearful), goosebumps (goosebumps)Disgusting (abominable), hate (hate), Dislike (dislike), unpleasant (displeased), hateful, annoying, irritating (annoying), embarrassed (embarrassed), arrogant (arrogant), angry (angry), furious (angry), indignant (infuriated), upset (get angry)

[0129] In one embodiment, the data may include various learning data and / or parameters acquired based on the user's learning through interaction with the user. In one embodiment, the data may include various schemas (or algorithms, models, networks, or functions) for supporting artificial intelligence-based operations.

[0130] For example, a scheme for supporting artificial intelligence-based operations in an electronic device (400) may include a neural network. In one embodiment, the neural network may include a neural network model based on at least one of an artificial neural network (ANN), a convolution neural network (CNN), a region with convolution neural network (R-CNN), a region proposal network (RPN), a recurrent neural network (RNN), a stacking-based deep neural network (S-DNN), a state-space dynamic neural network (S-SDNN), a deconvolution network, a deep belief network (DBN), a restricted Boltzman machine (RBM), a long short-term memory (LSTM) network, a classification network, a plain residual network, a dense network, a hierarchical pyramid network, and / or a fully convolutional network. According to one embodiment, the type of the neural network model is not limited to the examples described above.

[0131] According to one embodiment, the memory (420) may store instructions that, when executed, cause the processor (410) to operate. The memory (420) may store instructions that, when individually and / or collectively executed by the processor (410), cause the electronic device (400) to perform operations.

[0132] According to one embodiment, the memory (420) may store instructions that, when individually and / or collectively executed by the processor (410), cause the electronic device (400) to, in response to an input of a search term requesting a search for content related to an emotion (e.g., images and / or videos), extract emotional keywords from the search term, search for content within the electronic device (400) based on the emotional keywords, group the content based on time and place, generate a prompt (e.g., a text prompt) for inferring a cause of an emotion related to the emotional keyword based on the grouped content, provide (or transfer) the grouped content and prompt to a generative AI of the on-device and / or server, obtain data (or result data) (e.g., a description of the cause of the emotion) based on the grouped content and prompt, and display an interface including the grouped content and data based on the data on the display (440).

[0133] For example, the instructions may be stored as software (e.g., program (140) of FIG. 1) on the memory (420) and executable by the processor (410). For example, the instructions may include control commands such as arithmetic and logical operations, data movement, and / or input / output that may be recognized by the processor (410). According to one embodiment, the software may include various applications (e.g., application (146) of FIG. 1) that may provide various functions (or services) (e.g., search function, tag extraction function, content grouping function, prompt generation function, interactive service function, routine function, call function, message function, messenger function, e-mail function, SNS (social networking service) function, search function, media (e.g., video and / or music) playback function, game function, and / or wireless communication function) in the electronic device (400).

[0134] The display (440) may include a configuration identical or similar to that of the display module (160) of FIG. 1. The display (440) may display various images provided from the processor (410). Under the control of the processor (410), the display (440) may visually provide an application being executed (e.g., the application (146) of FIG. 1) and various screens related to its use (e.g., a contents screen, an application execution screen, a menu screen, and / or a function execution screen). According to one embodiment, the screen size of the display (440) may change depending on the form factor of the electronic device (400) (e.g., a bar-type smartphone (491-1), a foldable-type smartphone (491-2), a slider-type smartphone (491-3), a tablet (492)). For example, the display (440) may be configured to provide a first state having a first screen size and a second state having a second screen size larger than the first screen size.

[0135] The display (440) may be combined with a touch sensor, a pressure sensor capable of measuring the intensity of a touch, and / or a touch panel (e.g., a digitizer) that detects a magnetic stylus pen. The display (440) may detect a touch input, an air gesture input, and / or a hovering input (or a proximity input) by measuring a change in a signal (e.g., voltage, light intensity, resistance, electromagnetic signal, and / or charge amount) for a specific location of the display (440) based on the touch sensor, the pressure sensor, and / or the touch panel. For example, the display (440) may include a touch screen that detects a touch and / or a proximity touch (or a hovering) input using a part of a user's body (e.g., a finger) or an input device (e.g., a stylus pen).

[0136] The display (440) may include, but is not limited to, a liquid crystal display (LCD), a light-emitting diode (LED) display, an organic light-emitting diode (OLED) display, and / or an active matrix OLED (AMOLED) display, a micro electro mechanical systems (MEMS) display, or an electronic paper display. In one embodiment, the display (440) may include a flexible display.

[0137] The communication circuit (460) can support the establishment of a designated wireless communication channel (e.g., short-range communication such as Bluetooth communication and / or BLE communication) and the performance of communication through the established wireless communication channel. For example, the communication circuit (460) can perform designated communication (e.g., Bluetooth communication and / or BLE communication) with an external device. The communication circuit (460) can support wireless communication with an external device using cellular wireless communication (e.g., 4G LTE, 5G NR) and / or short-range wireless communication (e.g., Wi-Fi). For example, the electronic device (400) can communicate with an external server (e.g., a generative artificial intelligence server) that provides artificial intelligence-based functions (e.g., conversational services, assistant services, or AI agents) through a network using the communication circuit (460). According to one embodiment, the communication circuit (460) can transmit data generated in the electronic device (400) to the external server and receive data transmitted from the external server. The communication circuit (460) may include at least some of the configuration and / or functions of the communication module (190) of FIG. 1.

[0138] FIG. 5 is a diagram illustrating an example of a system and operation for providing emotional information using generative AI in an electronic device according to one embodiment of the present disclosure.

[0139] According to one embodiment, FIG. 5 may illustrate components of an exemplary system capable of performing the operations described herein. The components may be individually or collectively included in the electronic device (400), the processor (410) of the electronic device (400), and / or the server.

[0140] Referring to FIG. 5, according to one embodiment, the system may include components including a search request receiving unit (510), an emotional keyword extraction unit (515), a similar emotional-based search term generation unit (520), an emotional-based search term conversion unit (525), a content search engine (530), a database (DB) (540), a search result classification and content addition acquisition unit (550), a group place name determination and title generation unit (560), a prompt generation unit (570), a data generation unit (575), and / or an interface generation and provision unit (580). The above components are merely exemplary. For example, the system may include other components. For example, some components may be omitted from the electronic device (400). For example, some components may be integrated into one component.

[0141] In one embodiment, the search request receiving unit (510) (or user request receiving unit) may receive requests based on the user's emotions. For example, the search request receiving unit (510) may acquire (or receive) various multimodal inputs, such as the user's voice input, text input, and / or gesture input, as requests.

[0142] In one embodiment, the emotion keyword extraction unit (515) may analyze a user request and extract emotion keywords. In one embodiment, the emotion keywords may include emotion keywords (e.g., joy, happiness, sadness, and / or rewarding) from a whitelist that is predefined or arbitrarily added by a user (or a creator) if deemed useful. In one embodiment, the emotion keywords may be selected by inference from an AI model such as an LLM (large language model) in a description of a situation or emotional state (e.g., a user with a bright expression upon receiving a gift -> keywords such as happiness, joy, etc.). In one embodiment, for inference of emotion keywords, the use of prompts and operations for inferring emotion keywords contained in the emotion-based request received through an external AI model such as an LLM may be included. In one embodiment, the emotion keyword extraction unit (515) may perform an operation of generating and / or determining emotion keywords similar to the emotion keyword.

[0143] According to one embodiment, the emotion keyword extraction unit (515) can acquire a request based on the user's emotion by extracting artificial intelligence intents (AI intents) / entities (AI entities) through natural language processing (NLP). In one embodiment, an intent may represent a technical concept of NLP for identifying a user's intention or purpose. For example, in the question "Tell me the weather in Seoul today," the user's intention may be "requesting weather information," and an intent classification model may be used to identify this intent. In one embodiment, the intent classification model may include a model that receives text as input and classifies the intent of the input text. In one embodiment, an entity may represent a word or phrase representing a specific object or data. For example, in the question "Tell me the weather in Seoul today," "Seoul" may represent a regional entity, and "weather" may represent a weather entity. In one embodiment, an entity recognition model may recognize entities in text and extract necessary information based on the entities.

[0144] According to one embodiment, the emotion keyword extraction unit (515) can identify the purpose and intention related to the emotion included in the user's utterance or text input through NLP. For example, when the emotion keyword extraction unit (515) obtains a request such as "Find a picture that makes me happy in the past 6 months," it can obtain the purpose information such as "Find a picture" as an intent through NLP, etc., and obtain information such as "A picture that makes me happy in the past 6 months" as an entity. According to one embodiment, the emotion keyword extraction unit (515) can obtain a request that includes the user's emotion by extracting the emotion information of happiness from the entity, and recognizing the intent (e.g., "Find a picture") together with the emotion information of happiness.

[0145] In one embodiment, although not illustrated in FIG. 5, the system may include a detection unit for detecting similar tags to emotional keywords within the electronic device. For example, the detection unit may detect whether similar tags exist for content (e.g., images and / or videos) currently managed within the electronic device (400) in relation to the emotional keywords included in the user request.

[0146] According to one embodiment, the similar emotion-based search term generation unit (520) may be included in and operate as an AI model (e.g., generative AI) of the electronic device (400) and / or the server. According to one embodiment, the similar emotion-based search term generation unit (520) may generate a search term by converting it using a predetermined AI model (e.g., LLM) to derive accurate results for a user's request, such as a search for content expressing similar emotions, based on the user's emotion-based search request. According to one embodiment, the similar emotion-based search term generation unit (520) may separate metadata (e.g., time, space, and / or object (e.g., person, thing) information, etc.) from content (e.g., photos) among the emotion-based search requests through LLM or language pattern analysis. For example, if the search term 'a happy memory with A in California within the past 6 months' is entered, it may be analyzed as the time 'within 6 months', the place 'California', and the person 'A'. This process, commonly referred to as NER (name entity recognition), can be performed using pre-programmed language patterns or analyzed by LLM. Afterwards, in addition to the analyzed time and location information, a search can be attempted for photos of A that contain "my happiness" or "A's happiness."

[0147] According to one embodiment, the emotion-based search term conversion unit (525) (or emotion-based search term vector conversion unit) may perform an operation of vectorizing (e.g., converting into a vector value) the user's emotion-based search request text in order to perform a search within a DB (540) that is managed by converting into vectors of contents (e.g., images and / or videos) within the electronic device (400). According to one embodiment, vectorization may include an operation of converting meaning and context information for unstructured data such as images and / or videos into numerical representations. According to one embodiment, the emotion-based search term conversion unit (525) may perform a search through vector search on unstructured data (e.g., images and / or videos) stored as numerical information through vectorization, and may provide a similarity search function so as to search for values ​​that have similarity (or similarity) with respect to similar data and / or contextual meanings through vector values.

[0148] According to one embodiment, the emotion-based search term conversion unit (525) can configure a similarity search by adding various positive or negative vectors so that the search can be performed by understanding the overall context, rather than simply converting emotions into vector values ​​in order to perform a more accurate and personalized search for emotions in the vectorization operation. For example, when searching for the keyword 'happiness', if vector values ​​corresponding to a smiling face and firecrackers are present in the search image, the similarity score can be set higher. Here, the emotion-based search term conversion unit (525) can also configure a vector value by mixing in content corresponding to personalization. For example, if the personalization engine determines that the user dislikes the sound of firecrackers, the emotion-based search term conversion unit (525) can give a negative value to the vector for the keyword 'happiness' when firecrackers are included, thereby setting a low similarity score. According to one embodiment, a content search engine (530) can search for content (e.g., images and / or videos) within an electronic device (400).

[0149] According to one embodiment, the content search engine (530) may perform a similarity-based search (531) between a search term and a pre-stored emotion tag and / or a similarity-based search (533) between a search term vector and a pre-stored vector. According to one embodiment, the content search engine (530) may search for content through a search based on a similarity between a user-inputted emotion-based search and a content tag (e.g., a user-set tag and / or a hidden tag based on analysis of the electronic device (400)) pre-stored and managed in the electronic device (400). According to one embodiment, the content search engine (530) may also perform a content search by performing a search based on a similarity analysis between a search term vector value and vector values ​​of contents in a pre-stored user DB (540).

[0150] In one embodiment, similarity analysis of vector values ​​may be performed by performing a search on unstructured data (e.g., images and / or videos) through vector search (e.g., search via a content search engine (530)) using artificial intelligence learning (e.g., machine learning). For example, an artificial neural network (ANN) algorithm may be used to search for similar data (e.g., data containing synonyms, associations, and / or meaning behind the search) through vector values.

[0151] According to one embodiment, when the emotion-based search request text (e.g., search term) received from the user is transmitted to the content search engine (530) without converting the user's emotion-based search request text into a vector, the content search engine (530) can perform processing for search.

[0152] According to one embodiment, the database (DB) (540) may correspond to the memory (130) of FIG. 1, the memory (240) of FIG. 2, and / or the memory (420) of FIG. 4. According to one embodiment, the DB (540) may store and manage a plurality of content tags (e.g., emotion tags) according to a time point. The content tags may include user-set tags and / or hidden tags of various time points based on analysis of the electronic device (400). For example, the DB (540) may store and manage an emotion tag at the time of capturing an image (e.g., #A_emotion), an emotion tag at the time of analyzing the image (e.g., #B_emotion), an emotion tag at the time of checking the image (e.g., #C_emotion), and / or an emotion tag at the time of inputting a user tag (e.g., #D_emotion).

[0153] According to one embodiment, the search result classification and content addition acquisition unit (550) may process operations such as content grouping based on time and / or location (551) and time and / or location-based additional content acquisition (550). For example, the search result classification and content addition acquisition unit (550) may classify and group searched contents based on time and / or location by referring to each metadata information.

[0154] According to one embodiment, the search result classification and content addition acquisition unit (550) may group contents acquired at the same or similar places based on location information (e.g., place) at which the contents were acquired by referring to metadata information of the contents. According to one embodiment, the search result classification and content addition acquisition unit (550) may group contents acquired at the same or similar time zone based on time information at which the contents were acquired by referring to metadata information of the contents. According to one embodiment, the search result classification and content addition acquisition unit (550) may also classify and group contents acquired at the same or similar places at the same or similar time zone by considering both time and place.

[0155] For example, the search result classification and content addition acquisition unit (550) can identify which location the content corresponds to (e.g., identify a location indicated by an object in the content based on image analysis) through content analysis (e.g., image analysis) of the searched content (e.g., images or videos). According to one embodiment, the search result classification and content addition acquisition unit (550) can acquire location information of the searched content using a scene analysis AI model. According to one embodiment, the search result classification and content addition acquisition unit (550) can perform grouping based on the analyzed location information for the content.

[0156] According to one embodiment, the search result classification and content addition acquisition unit (550) can identify a predetermined condition for the contents within each group. According to one embodiment, the search result classification and content addition acquisition unit (550) can identify whether the number of contents within a group has a condition exceeding a threshold value and / or whether the basis data (or amount of information) for explaining emotions from the contents within the group (e.g., object information included in the contents) has a condition exceeding a threshold value. According to one embodiment, if the search result classification and content addition acquisition unit (550) determines that the number of contents within a group is less than a threshold value and / or the basis data for explaining emotions from the searched content information is insufficient (e.g., when object information included in the contents does not satisfy a predetermined condition), the search for additional content can be performed. For example, if an image is collected from the searched content that does not show a person's facial expression, or an image is collected for which it is determined that it is not easy to create a description in an AI model because the number of objects included in the image is below a standard, the search result classification and content addition acquisition unit (550) may search for additional content that may be included in the group based on the characteristics of the searched content within the group (e.g., characteristics of the image (e.g., time, place, and / or object, etc.)) and / or the user search request text.

[0157] According to one embodiment, the search result classification and content addition acquisition unit (550) can add content to a group created through a search based on the criteria information (e.g., location information, time information, and / or object information) that created the group. According to one embodiment, the search result classification and content addition acquisition unit (550) can group the content based on criteria information such as location information, time information, and / or object information (e.g., similarity information of objects in an image) from which the content is acquired when creating a group. In one embodiment, the criteria information related to the grouping of the content is not limited to the above-described example, and may include various criteria information that can be used for grouping, and a group may be created by one or more criteria information, or content may be added based on the criteria information.

[0158] According to one embodiment, the search result classification and content addition acquisition unit (550) can identify time information and location information when capturing content grouped based on a search term if the content includes metadata information (or metadata). In one embodiment, the location information is composed of a GPS coordinate system provided by the electronic device (400) and can indicate latitude [GPSLatitude], longitude [GPSLlongitude], and altitude [GPSAltitude] values. In one embodiment, in order to add other contents other than the contents grouped in advance based on location information, a range can be set by applying a specific threshold value to the latitude, longitude, and altitude based on the median of the latitude and longitude of the grouped contents. In one embodiment, if the average value of the latitude and longitude is used, the location error range may increase due to outliers that greatly deviate from the location value, making it difficult to reflect the location information of the existing grouped contents. Therefore, the median value that is insensitive to outliers can be used.

[0159] In one embodiment, when adding content to a group using time information in addition to location information, a time threshold may also be applied. For example, if content is added based solely on time information, content acquired (e.g., photographed) from other locations within the time threshold range may also be included in groups not intended by the user. To prevent this issue, content can be added using both time and location information as conditions.

[0160] According to one embodiment, the group place name determination and title generation unit (560) can determine the exact name of the place (e.g., XX Market) based on metadata information (e.g., location information) of the contents within each group and in conjunction with a designated application (e.g., a map application). According to one embodiment, the group place name determination and title generation unit (560) can identify objects of the contents within each group (e.g., identification through text OCR and / or object image identification through image analysis) to determine a more specific place name for the place. For example, the group place name determination and title generation unit (560) can determine the main place name (e.g., XX Market) and the specific place name for the main place (e.g., 'OOOO' located within XX Market) through the place name determined in conjunction with the designated application and object identification included in the contents of the group. According to one embodiment, the group place name determination and title generation unit (560) may determine the title of the group (e.g., Paris Travel) based on place information (e.g., Paris, France) and / or image scene analysis (e.g., suitcase, airplane, airport, Eiffel Tower, etc.).

[0161] In one embodiment, the group location name determination and title generation unit (560) can perform image captioning (or scene analysis). For example, the group location name determination and title generation unit (560) can use a captioning LLM that receives content as input, analyzes it, and outputs the status of the content in text format to generate a title for the grouped content. In one embodiment, the captioning LLM can include a contrastive language-image pre-training (CLIP) model and a bootstrapping language-image pre-training (BLIP) model. For example, the group location name determination and title generation unit (560) can use the BLIP2 model for image captioning. In one embodiment, the BLIP2 model has better zero-shot learning (ZSL) performance than other models and can be frequently used for image captioning. In one embodiment, zero-shot learning performance can be evaluated based on how accurately the AI ​​model outputs results when using content (e.g., images and / or videos) that is not included in the training data used to train the LLM as input.

[0162] In one embodiment, the image captioning LLM can use data in which images (e.g., input data) and caption information describing the images (e.g., output) are paired. In one embodiment, datasets used for model training may include datasets such as COCO (common objects in context), Visual Genome, CC3M (conceptual captions 3 million), CC12M (conceptual captions 12 million), SBU (Stanford-Bing University), and / or LAION-400M. In one embodiment, pseudo-captioning may be performed on images collected online using an algorithm such as CapFlip, and used as training data for the captioning LLM. In one embodiment, data sets such as COCO, Visual Genome, CC3M, CC12M, SBU, and / or LAION-400M may be constructed by inputting text information about images in advance, and for various types of images or a large number of images for which no text information has been input, images collected online may be used.

[0163] According to one embodiment, the prompt generation unit (570) (e.g., a prompt generation unit for generating an emotion-based description) may include an image representation AI model (e.g., an image-to-text conversion AI model, a multimodal LLM, etc.). According to one embodiment, the prompt generation unit (570) may generate a prompt for generating a description (e.g., emotional information) of searched contents using an AI model (e.g., a generative AI). According to one embodiment, the prompt generation unit (570) may also generate a prompt using a predefined template for a prompt that is passed as an input to an AI model for generating a content-based description. For example, in order to utilize 'emotion keywords extracted from a search request input by a user, a data field that may include keywords for emotions may be defined, and a prompt with the content of 'generating a description that infers the cause that may cause the corresponding emotion through descriptions of contents input to the AI ​​model' may be generated.

[0164] In one embodiment, the prompt generation unit (570) can generate a prompt that can describe an emotion using a template. In one embodiment, when generating a prompt, the prompt generation unit (570) can utilize additional information (e.g., user-related information) in addition to the user's core keywords as information (e.g., prompt source) for generating the prompt. In one embodiment, the user-related information may include, for example, information such as the user's characteristics, age, age, pattern, and gender. For example, the prompt generation unit (570) can generate a personalized prompt using the user-related information.

[0165] In one embodiment, when a user conducts an emotion-based search, text describing the emotion can be used as a fixed input for prompt generation as a core keyword for describing grouped content. In one embodiment, the prompt generation unit (570) can use a prompt template to identify the cause of the emotion searched by the user in the grouped content using the search emotion keyword. For example, when the word (or tag) "happiness" is searched in a gallery, the prompt template for generating a description can be structured in the format "Describe how the keyword (e.g., "happiness") feels, considering images (e.g., Group 1)." In one embodiment, because the search is conducted based on emotion and the description is generated based on the search, the template as exemplified can accurately reflect the user's intent. In one embodiment, using a template can reduce time complexity compared to generating a prompt to describe an emotion every time a user conducts an emotion search. In one embodiment, using a template can generate a fixed and more accurate prompt for describing an emotion.

[0166] In one embodiment, the natural language input by the user or the template described above may be referred to as a prompt, and when the prompt is passed as input to the LLM or generative model, vector embedding (or vectorization) may be performed. For example, when vector embedding is performed, each word is composed of a real number of a one-dimensional vector, and the size of the vector may vary depending on the length of the natural language. In one embodiment, the prompt input by the user or generated by the electronic device (400) may first undergo tokenization. In one embodiment, a token may mean a basic unit of information that the LLM reads to understand and generate text. In one embodiment, a token may be composed of an individual character, an entire word, or a part of a word. For example, given a natural language input (e.g., “This article rocks!”), tokenization is first performed, and then vector embedding (e.g., “[576, 4652, 23902]”) is performed on each tokenized unit to convert it into information recognizable by the LLM, which can then be used as input to the LLM. In one embodiment, tokenization allows the tokenizer to semantically distinguish tokens from individual characters or words. For example, the tokenizer can classify the tokens in the word “Evergreen” into “Ever” and “green.”

[0167] According to one embodiment, the data generation unit (575) (e.g., group-specific emotion-based content description generation unit) may operate based on an AI model (e.g., generative AI). According to one embodiment, the data generation unit (575) inputs a prompt to generate a description explaining the cause of the emotion based on the determined contents and emotions within the group to the generative AI (e.g., LLM or an image representation AI model, etc.), so that an explanation explaining the cause of the emotion may be generated and output based on the contents and emotion keywords within the group. For example, the data generation unit (575) may infer the cause of the emotion through the content and emotion keywords and generate an explanation (e.g., emotion information). For example, if a “Paris trip” is identified through scene analysis of an image and “happiness” is identified as an emotion keyword, the explanation “I am happy because of the pleasant trip to Paris” may be generated.

[0168] In one embodiment, the description generation may utilize LLM. For example, the LLM for generating the final description for the grouped contents may use the same model as the LLM used for the group title generation. In one embodiment, only the grouped contents may be input to the LLM for title generation without separate user prompt input. For content description generation, the existing LLM may use the content along with user text (e.g., sentiment search keywords + templates), and the LLM output may be identical in text form.

[0169] According to one embodiment, the interface generation and provision unit (580) may group results (or contents or result data) in response to a user's search request according to reference information (e.g., timeline, location, person, object, weather, etc.), and may sort and display the grouped contents according to the reference information and provide them. For example, the interface generation and provision unit (580) may group and display contents according to various reference information when providing an interface. For example, the interface generation and provision unit (580) may display emotion-based descriptions for all searched and grouped contents in a summary form in an interface (e.g., a second interface) that displays the results. For example, when a user selects a graphic element (e.g., a content object or an image object) representing each group in the second interface, the interface generation and provision unit (580) may display and provide descriptions explaining the emotion-based cause only for contents included in the corresponding group. For example, the interface creation and provision unit (580) can switch from the second interface to the first interface and display a description explaining the cause based on emotion only for contents included in a group selected by the user.

[0170] An electronic device (e.g., an electronic device (101, 201, 400) of FIG. 1, 2, or 4) according to one embodiment of the present disclosure (hereinafter referred to as electronic device (101)) may include a display (e.g., a display module (160) of FIG. 1 or a display (440) of FIG. 4) (hereinafter referred to as display (160)), at least one processor (e.g., a processor (120, 230, 410) of FIG. 1, 2, or 4) including processing circuitry (hereinafter referred to as processor (120)), and a memory (e.g., a memory (130, 240, 420) of FIG. 1, 2, or 4) (hereinafter referred to as memory (130)). In one embodiment, the memory (130) may store instructions that, when individually and / or collectively executed by the processor (120), cause the electronic device (101) to perform operations.

[0171] According to one embodiment, the instructions, when individually and / or collectively executed by at least one processor (120), may cause the electronic device (101) to receive an input of a search term requesting a content search related to an emotion. The instructions, when individually and / or collectively executed by at least one processor (120), may cause the electronic device (101) to extract an emotion keyword from the search term in response to the input. The instructions, when individually and / or collectively executed by at least one processor (120), may cause the electronic device (101) to perform a content search within the electronic device based on the emotion keyword. The instructions, when individually and / or collectively executed by at least one processor (120), may cause the electronic device (101) to group searched content based on time and location. The instructions, when individually and / or collectively executed by at least one processor (120), may cause the electronic device (101) to generate a prompt for inferring a cause of an emotion related to the emotion keyword based on the group content. The instructions, when individually and / or collectively executed by at least one processor (120), may cause the electronic device (101) to transmit the group content and the prompt to an on-device and / or server-generated artificial intelligence (AI). The instructions, when individually and / or collectively executed by at least one processor (120), may cause the electronic device (101) to acquire data based on the group content and the prompt.The above instructions, when individually and / or collectively executed by at least one processor (120), may cause the electronic device (101) to display an interface including the group content and data on a display based on the data.

[0172] According to one embodiment, the instructions, when individually and / or collectively executed by at least one processor (120), may cause the electronic device (101) to determine whether the retrieved content satisfies a given condition. The instructions, when individually and / or collectively executed by at least one processor (120), may cause the electronic device (101) to search for similar content by group based on time and / or location related to content within a group that does not satisfy the given condition, if the retrieved content does not satisfy the given condition.

[0173] According to one embodiment, the instructions, when individually and / or collectively executed by at least one processor (120), may cause the electronic device (101) to generate the prompt based on whether the retrieved content has a defined condition.

[0174] According to one embodiment, the instructions, when individually and / or collectively executed by at least one processor (120), may cause the electronic device (101) to generate a first prompt for inferring a cause of an emotion related to an emotion keyword based on content by group, transmit the content by group and the first prompt as inputs to a first generative AI, and obtain emotion-based data (e.g., first data) for each content within the group. The instructions, when individually and / or collectively executed by at least one processor (120), may cause the electronic device (101) to generate a second prompt for generating a summary based on the first data and the emotion keyword of all groups, transmit the second prompt as inputs to a second generative AI, and obtain emotion-based data (e.g., second data) for each group.

[0175] In one embodiment, the first data may be obtained from the first generative AI and may include an emotion-based description of the content within each group. In one embodiment, the second data may be obtained from the second generative AI and may include an emotion-based description of the summary of each group.

[0176] In one embodiment, the first generative AI and the second generative AI may be one generative AI or may include separate generative AIs.

[0177] According to one embodiment, the instructions, when individually and / or collectively executed by at least one processor (120), may cause the electronic device (101) to display a second interface based on the representative content and the second data for each group.

[0178] According to one embodiment, the instructions, when individually and / or collectively executed by at least one processor (120), may cause the electronic device (101) to receive an input for selecting a group based on the second interface. The instructions, when individually and / or collectively executed by at least one processor (120), may cause the electronic device (101) to, in response to the input, display a first interface based on content and first data related to the selected group.

[0179] According to one embodiment, the instructions, when individually and / or collectively executed by at least one processor (120), may cause the electronic device (101) to display each group in an order based on a timeline interface.

[0180] In one embodiment, the interface may include a representative emotional image for the entire content based on the data.

[0181] According to one embodiment, the instructions, when individually and / or collectively executed by at least one processor (120), may cause the electronic device (101) to identify a representative emotion regarding a topic of the entire content based on the data. The instructions, when individually and / or collectively executed by at least one processor (120), may cause the electronic device (101) to generate an emotional image representing the representative emotion. The instructions, when individually and / or collectively executed by at least one processor (120), may cause the electronic device (101) to display the emotional image by placing it on the interface.

[0182] According to one embodiment, the instructions, when individually and / or collectively executed by at least one processor (120), may cause the electronic device (101) to infer an emotion based on an emotion analysis of content within the electronic device and / or to identify an emotion based on an emotion tag set in content within the electronic device. The instructions, when individually and / or collectively executed by at least one processor (120), may cause the electronic device (101) to search for content related to the emotion keyword based on the inference and / or identification related to the content.

[0183] Hereinafter, an operating method of an electronic device (101, 201, 400) according to various embodiments (hereinafter referred to as electronic device (101)) will be described in detail. Operations performed in the electronic device (101) according to various embodiments may be executed by at least one processor (120, 230, 410) (hereinafter referred to as processor (120)) including various processing circuitry and / or executable program elements of the electronic device (101). According to one embodiment, the operations performed in the electronic device (101) may be stored as instructions in a memory (130, 240, 420) (hereinafter referred to as memory (130)) and individually and / or collectively performed (or executed) by the processor (120).

[0184] FIG. 6 is a flowchart illustrating a method of operating an electronic device according to one embodiment of the present disclosure.

[0185] According to one embodiment, FIG. 6 may illustrate an example of a method for providing an emotion cause description (e.g., emotion information) using emotion tag-based content (e.g., image and / or video) search and generative AI in an electronic device (101) according to one embodiment.

[0186] An operation method performed in an electronic device (101) according to one embodiment of the present disclosure may be performed, for example, according to a flowchart illustrated in FIG. 6. The flowchart illustrated in FIG. 6 is an example according to one embodiment of an operation of the electronic device (101), and the order of at least some operations may be changed, performed in parallel, performed as independent operations, or at least some other operations may be performed complementarily to at least some operations. According to one embodiment of the present disclosure, operations 601 to 615 may be performed in at least one processor (120) of the electronic device (101).

[0187] As illustrated in FIG. 6, an operation method performed by an electronic device (101) according to one embodiment may include an operation of receiving an input of a search word requesting a content search related to an emotion (operation 601), an operation of extracting an emotion keyword from the search word (operation 603), an operation of performing a content search within the electronic device (101) (operation 605), an operation of grouping content based on predetermined reference information (operation 607), an operation of generating a prompt for inferring a cause of an emotion related to an emotion keyword based on content by group (operation 609), an operation of transmitting content and a prompt by group to a generative AI (operation 611), an operation of acquiring emotion-based data by group (operation 613), and an operation of displaying an interface including content and data by group (operation 615).

[0188] Referring to FIG. 6, in operation 601, the processor (120) of the electronic device (101) may receive an input of a search word requesting a search for content related to emotions. According to one embodiment, the processor (120) may display an execution screen of an application related to content search (e.g., a search application or a gallery application) on the display (160). According to one embodiment, the processor (120) may receive an input related to application execution from a user. According to one embodiment, in response to the input related to application execution, the processor (120) may execute the application and display the execution screen of the application on the display (160). According to one embodiment, the processor (120) may receive an input of a search word from the user while displaying the execution screen. According to one embodiment, the processor (120) may receive a text-based search word input entered through a keypad of the execution screen and / or a voice-based search word input entered through a microphone. In one embodiment, the search term may include emotion keywords related to emotions (e.g., the emotion terms exemplified in Table 1). In one embodiment, the processor (120) may, in response to the input of the search term, determine whether to perform an emotion tag-based content search for content within the electronic device (101) related to emotions. In one embodiment, the content search may include, for example, a context search using vector search.

[0189] In operation 603, the processor (120) may extract emotional keywords from the search term in response to input of a search term requesting a search for content related to emotions. In one embodiment, the processor (120) may extract emotional keywords related to emotions (e.g., terms corresponding to representative emotions and / or similar emotions as exemplified in ) from the search term (e.g., text characters).

[0190] In operation 605, the processor (120) may perform a content search within the electronic device (101). According to one embodiment, the processor (120) may search for content within the electronic device (101) in which an emotion tag (e.g., a user-defined tag and / or a device analysis-based hidden tag) corresponding to the emotion keyword is set based on the emotion keyword. According to one embodiment, the processor (120) may infer an emotion based on emotion analysis of the content within the electronic device (101) and / or identify an emotion based on an emotion tag set for the content within the electronic device (101). According to one embodiment, the processor (120) may search for content related to the emotion keyword based on inference and / or identification related to the content.

[0191] According to one embodiment, the processor (120) may perform a content search within the electronic device (101) using a search term-based vector conversion value. For example, the processor (120) may perform an operation of vectorizing (e.g., converting into a vector value) an emotion-based search term (e.g., text characters) to perform a search within a DB (540) that converts and manages content (e.g., images and / or videos) within the electronic device (400) into vectors.

[0192] According to one embodiment, the processor (120) may perform a similarity search by adding various positive or negative vectors so that the search can be performed by grasping the overall context, rather than simply converting emotions into vector values ​​in order to perform a more accurate and personalized search for emotions in the vectorization operation. For example, when searching for the keyword 'happiness', if vector values ​​corresponding to a smiling face and firecrackers exist in the search content, the similarity score may be set higher. According to one embodiment, in relation to the vector search, when the processor (120) converts an emotion-based search term (e.g., text characters) into a vector value, the processor (120) may perform a search for a corresponding vector value in a pre-stored vector DB (e.g., DB (540) of FIG. 5) using the converted vector value.

[0193] In operation 607, the processor (120) may group the content based on predetermined criteria information (e.g., time, place, person, and / or object, etc.). In one embodiment, the content may include content (e.g., images and / or videos, etc.) searched by the search operation of operation 605. According to one embodiment, if the processor (120) determines that the number of contents in each group is less than or equal to a predetermined number or that it is impossible to generate an emotion-based cause explanation (e.g., emotion information) using an AI model (e.g., an image representation AI model or a generative AI) using the content, the processor (120) may additionally search and acquire similar contents based on the criteria information in each group, and group the contents based on the predetermined criteria information, including the primarily acquired contents and the secondarily acquired contents.

[0194] In operation 609, the processor (120) may generate a prompt for inferring the cause of an emotion related to an emotion keyword based on content for each group. In one embodiment, the processor (120) may generate a prompt (e.g., a text prompt) for inferring the cause of an emotion related to an emotion keyword based on content for each group. In one embodiment, the processor (120) may generate a prompt including a search term and an emotion keyword to infer the cause of an emotion related to an emotion keyword for content for each group.

[0195] According to one embodiment, the processor (120) may generate a text prompt (or instruction) to provide content and a description (or emotional information) corresponding to the search term based on a search term input. According to one embodiment, the processor (120) may also generate the prompt using a predefined template for a prompt that is passed as an input to an AI model for generating a content-based description (or emotional information). For example, in order to utilize 'emotion keywords extracted from a search term input by a user, a data field that may include keywords for emotions may be defined, and a prompt with the content of 'generating a description that infers the cause that may cause the corresponding emotion through descriptions of contents input to the AI ​​model' may be generated.

[0196] In operation 611, the processor (120) may transmit group-specific content and prompts to a generative AI. In one embodiment, the processor (120) may transmit (or provide) the group-specific content and prompts to a generative AI (artificial intelligence) model on-device and / or on a server. In one embodiment, the processor (120) may provide (or provide) prompts to the generative AI model so that data (e.g., a description or emotional information) is provided based on the prompts.

[0197] In one embodiment, the prompt may be provided to the on-device generative AI model and / or the server generative AI model (e.g., LLM or Image Representation AI model). In one embodiment, the data may be provided based on the on-device and / or the server. In one embodiment, the processor (120) may transmit the content and prompt for each group as input to the generative AI model (e.g., LLM or Image Representation AI model, etc.) so as to generate an emotion-based first description for each group. In one embodiment, the processor (120) may transmit the descriptions for all groups and a prompt for generating a summary based on emotion keywords to the generative AI so as to generate an emotion-based second description (e.g., summary) for the entire content.

[0198] In operation 613, the processor (120) may obtain group-specific emotion-based data. In one embodiment, the processor (120) may obtain data based on group-specific content and prompts. In one embodiment, the processor (120) may obtain data including a description from a generative AI model based on the prompt. In one embodiment, the processor (120) may obtain (or generate) data according to a description generation process (e.g., a description generation operation) executed in relation to the prompt in an on-device AI model. In one embodiment, the processor (120) may obtain (or receive) data according to a description generation process (e.g., a description generation operation) executed in relation to the prompt in a server AI model from a server.

[0199] According to one embodiment, the processor (120) may generate a first prompt for inferring the cause of an emotion related to an emotion keyword based on content for each group, and may transmit the content for each group and the first prompt as inputs to a first generative AI to obtain emotion-based data (e.g., first data) for each content within the group. According to one embodiment, the processor (120) may generate a second prompt for generating a summary based on the first data and emotion keywords for all groups, and may transmit the second prompt as inputs to a second generative AI to obtain emotion-based data (e.g., second data) for each group. In one embodiment, the first data may be obtained from the first generative AI and may include an emotion-based description of the content within each group. In one embodiment, the second data may be obtained from the second generative AI and may include an emotion-based description of the summary of each group. In one embodiment, the first generative AI and the second generative AI may be a single generative AI, or may include separate generative AIs.

[0200] In operation 615, the processor (120) may display an interface including content and data by group. According to one embodiment, the processor (120) may display a designated interface including content and data (e.g., descriptions) by group on the display (160) based on the data. According to one embodiment, the processor (120) may provide an interface in which content and descriptions are connected in a hierarchical structure. According to one embodiment, the processor (120) may display content by group based on time-series characteristics. According to one embodiment, the processor (120) may sort and display each group based on a timeline interface. According to one embodiment, the interface is not limited to a timeline-based interface, and the interface may also sort and provide groups based on predetermined reference information (e.g., timeline, location, person, object, and / or weather, etc.).

[0201] According to one embodiment, the processor (120) may display a second interface (e.g., see FIG. 13) on the display (160) based on the representative content and second data (e.g., the emotion-based second description) for each group. According to one embodiment, while displaying the second interface, the processor (120) may receive an input for selecting a group based on the second interface. For example, the processor (120) may receive an input for selecting a portion corresponding to any one of the groups displayed through the second interface. According to one embodiment, in response to the input for selecting a group, the processor (120) may display a first interface (e.g., see FIG. 14) on the display (160) based on the content and first data (e.g., the emotion-based first description) related to the selected group (e.g., switching the second interface to the first interface).

[0202] In one embodiment, the interface (e.g., the first interface and / or the first interface) may include a representative emotional image (or emotional indicator) regarding the entire content based on data. In one embodiment, the processor (120) may identify a representative emotional image regarding the topic of the entire content based on the data. In one embodiment, the processor (120) may generate an emotional image representing the representative emotional image. In one embodiment, the processor (120) may place the emotional image on the interface and display it along with the content.

[0203] FIG. 7 is a flowchart illustrating a method of operating an electronic device according to one embodiment of the present disclosure.

[0204] According to one embodiment, FIG. 7 may illustrate an example of a method for providing an emotion cause description (e.g., emotion information) using emotion tag-based content (e.g., image and / or video) search and generative AI in an electronic device (101) according to one embodiment.

[0205] An operation method performed in an electronic device (101) according to one embodiment of the present disclosure may be performed, for example, according to a flowchart illustrated in FIG. 7. The flowchart illustrated in FIG. 7 is an example according to one embodiment of an operation of the electronic device (101), and the order of at least some operations may be changed, performed in parallel, performed as independent operations, or at least some other operations may be performed complementarily to at least some operations. According to one embodiment of the present disclosure, operations 701 to 709 may be performed in at least one processor (120) of the electronic device (101).

[0206] According to one embodiment, the operations described in FIG. 7 may be performed heuristically, for example, in combination with the operations described in FIG. 6, or heuristically performed as a replacement for at least some of the operations described and combined with at least some other operations, or heuristically performed as a detailed operation of at least some of the operations described.

[0207] As illustrated in FIG. 7, an operation method performed by an electronic device (101) according to an embodiment may include an operation of performing a content search within the electronic device (101) (operation 701), an operation of grouping contents based on defined reference information (operation 703), an operation of determining whether a defined condition is satisfied (operation 705), an operation of generating a prompt for inferring a cause of emotion related to an emotion keyword based on contents for each group if the defined condition is satisfied (operation 709), an operation of obtaining and grouping similar contents if the defined condition is not satisfied (operation 707), and an operation of generating a prompt for inferring a cause of emotion related to an emotion keyword based on contents for each group (operation 709).

[0208] Referring to FIG. 7, in operation 701, the processor (120) of the electronic device (101) may perform a content search within the electronic device (101). According to one embodiment, the processor (120) may search for content in which an emotion tag (e.g., a user-defined tag and / or a device analysis-based hidden tag) corresponding to the emotion keyword is set among the content within the electronic device (101) based on the emotion keyword. According to one embodiment, the processor (120) may infer an emotion based on emotion analysis of the content within the electronic device (101) and / or identify an emotion based on an emotion tag set for the content within the electronic device (101). According to one embodiment, the processor (120) may search for content related to the emotion keyword based on inference and / or identification related to the content.

[0209] According to one embodiment, the processor (120) may perform a content search within the electronic device (101) using a search term-based vector conversion value. For example, the processor (120) may perform an operation of vectorizing (e.g., converting into a vector value) an emotion-based search term (e.g., text characters) to perform a search within a DB (540) that converts and manages content (e.g., images and / or videos) within the electronic device (400) into vectors.

[0210] According to one embodiment, the processor (120) may perform a similarity search by adding various positive or negative vectors so that the search can be performed by grasping the overall context, rather than simply converting emotions into vector values ​​in order to perform a more accurate and personalized search for emotions in the vectorization operation. For example, when searching for the keyword 'happiness', if vector values ​​corresponding to a smiling face and firecrackers exist in the search content, the similarity score may be set higher. According to one embodiment, in relation to the vector search, when the processor (120) converts an emotion-based search term (e.g., text characters) into a vector value, the processor (120) may perform a search for a corresponding vector value in a pre-stored vector DB (e.g., DB (540) of FIG. 5) using the converted vector value.

[0211] In operation 703, the processor (120) may group the content based on defined criteria information (e.g., time, place, person, and / or object, etc.). In one embodiment, the content may include content (e.g., images and / or videos, etc.) searched by the search operation of operation 605. According to one embodiment, if the processor (120) determines that the number of contents in each group is less than or equal to a predetermined number or that it is impossible to generate an emotion-based cause explanation (e.g., emotion information) using an AI model (e.g., an image representation AI model or a generative AI) using the content, the processor (120) may additionally search and acquire similar contents based on the criteria information in each group, and group the contents based on the defined criteria information, including the primarily acquired contents and the secondarily acquired contents.

[0212] In operation 705, the processor (120) may determine whether a predetermined condition is satisfied. According to one embodiment, the processor (120) may identify whether the number of contents within a group exceeds a threshold and / or whether the basis data (or amount of information) for explaining emotions from the contents within the group (e.g., object information included in the contents) exceeds a threshold.

[0213] In operation 705, if the processor (120) determines that a predetermined condition is satisfied (e.g., 'yes' in operation 705), in operation 709, a prompt may be generated to infer the cause of the emotion related to the emotion keyword based on the content by group. According to one embodiment, the processor (120) may generate a prompt for generating a description (e.g., emotion information) of the searched contents using an AI model (e.g., generative AI). According to one embodiment, the processor (120) may also generate the prompt using a predefined template for the prompt that is transmitted as an input to an AI model for generating a content-based description. According to one embodiment, when generating the prompt, the processor (120) may utilize additional information (e.g., user-related information) in addition to the user's core keyword as information (e.g., prompt source) for generating the prompt. In one embodiment, the user-related information may include, for example, information such as the user's characteristics, age, age, pattern, and gender. For example, the processor (120) can generate personalized prompts using user-related information.

[0214] In operation 705, if the processor (120) determines that the predetermined condition is not satisfied (e.g., 'No' in operation 705), in operation 707, similar contents may be acquired and grouped. According to one embodiment, if the processor (120) determines that the number of contents in each group is less than or equal to a predetermined number and / or that it is impossible to generate an emotion-based cause explanation (e.g., emotion information) using an AI model (e.g., an image representation AI model or a generative AI) using the corresponding contents, time / location-based similar contents in each group may be additionally searched and acquired, and grouped based on reference information, including the primarily acquired contents and the secondarily acquired contents.

[0215] According to one embodiment, if the number of contents within a group is below a threshold value and / or the retrieved content information lacks sufficient basis data for explaining emotions (e.g., object information included in the content does not satisfy a predefined condition), the processor (120) may perform a search for additional contents (e.g., similar contents). For example, if an image without a human facial expression is collected from the retrieved contents, or an image is collected for which the number of objects included in the image is below a threshold value and therefore it is determined that it is not easy for an AI model to generate a description, the processor (120) may perform a search for additional contents (e.g., contents having characteristics identical to or similar to the characteristics of the contents) that may be included in the group based on the characteristics of the retrieved contents within the group (e.g., characteristics of the image (e.g., time, location, and / or object, etc.)) and / or a user search request text.

[0216] According to one embodiment, the processor (120) may add content to a group created through a search based on reference information (e.g., location information, time information, and / or object information) that created the group. According to one embodiment, when creating a group, the processor (120) may group the content based on reference information such as location information, time information, and / or object information (e.g., similarity information of objects in an image) from which the content is acquired. In one embodiment, the reference information related to grouping of content is not limited to the above-described example, and may include various reference information that can be used for grouping, and a group may be created by one or more reference information, or content may be added based on the reference information.

[0217] In operation 709, the processor (120) may generate a prompt for inferring the cause of an emotion related to an emotion keyword based on group-specific content. In one embodiment, the processor (120) may generate the prompt as an operation corresponding to a previous prompt generation operation (e.g., operation 709 following operation 707).

[0218] FIG. 8 is a flowchart illustrating a method of operating an electronic device according to one embodiment of the present disclosure.

[0219] According to one embodiment, FIG. 8 may illustrate an example of a method for providing an emotion cause description (e.g., emotion information) using emotion tag-based content (e.g., image and / or video) search and generative AI in an electronic device (101) according to one embodiment.

[0220] An operation method performed in an electronic device (101) according to one embodiment of the present disclosure may be performed, for example, according to a flowchart illustrated in FIG. 8. The flowchart illustrated in FIG. 8 is an example according to one embodiment of an operation of the electronic device (101), and the order of at least some operations may be changed, performed in parallel, performed as independent operations, or at least some other operations may be performed complementarily to at least some operations. According to one embodiment of the present disclosure, operations 801 to 817 may be performed in at least one processor (120) of the electronic device (101).

[0221] According to one embodiment, the operations described in FIG. 8 may be heuristically performed in combination with the operations described in FIGS. 6 to 7, for example, or may be heuristically performed as a replacement for at least some of the operations described and combined with at least some other operations, or may be heuristically performed as a detailed operation of at least some of the operations described.

[0222] As illustrated in FIG. 8, an operation method performed by an electronic device (101) according to an embodiment may include an operation of generating a first prompt for inferring a cause of an emotion related to an emotion keyword based on content by group (operation 801), an operation of transmitting the content by group and the first prompt as input to a generative AI (operation 803), an operation of obtaining emotion-based first data by content within a group (operation 805), an operation of generating a second prompt for generating a summary based on the first data and emotion keywords of all groups (operation 807), an operation of transmitting the second prompt as input to the generative AI (operation 809), an operation of obtaining emotion-based second data by group (operation 811), an operation of displaying a representative content by group and a second interface based on the second data (operation 813), an operation of receiving an input for selecting a group based on the second interface (operation 815), and an operation of displaying content related to the selected group and a first interface based on the first data (operation 817).

[0223] Referring to FIG. 8, in operation 801, the processor (120) of the electronic device (101) (e.g., the prompt generation unit (570) of FIG. 5) may generate a first prompt for inferring the cause of an emotion related to an emotion keyword based on content by group. According to one embodiment, the processor (120) may generate a first prompt that causes a description of the searched contents to be generated (e.g., a first description based on emotion by each group). According to one embodiment, the processor (120) may also generate the prompt using a predefined template for the first prompt that is transmitted as an input to an AI model for generating a content-based description.

[0224] In operation 803, the processor (120) (e.g., the prompt generation unit (570) of FIG. 5) may transmit the group-specific content and the first prompt as inputs to the generative AI. According to one embodiment, the processor (120) may provide the first prompt to the generative AI model of the on-device and / or the generative AI model of the server (e.g., an LLM or an Image Representation AI model). According to one embodiment, the processor (120) may transmit the group-specific content and the first prompt as inputs to the generative AI model (e.g., an LLM or an Image Representation AI model, etc.) so as to generate an emotion-based first description for each group.

[0225] In operation 805, the processor (120) may obtain first data based on emotion for each content within the group. According to one embodiment, the processor (120) may obtain the first data based on the content for each group and the first prompt. According to one embodiment, the processor (120) may obtain first data including a description from a generative AI model based on the first prompt. According to one embodiment, the processor (120) may obtain (or generate) first data according to a description generation process (e.g., a first description generation operation) executed in relation to the prompt in an on-device AI model. According to one embodiment, the processor (120) may obtain (or receive) first data according to a description generation process (e.g., a first description generation operation) executed in relation to the first prompt in a server AI model from a server.

[0226] According to one embodiment, the processor (120) generates a first prompt for inferring the cause of an emotion related to an emotion keyword based on content by group, and transmits the content by group and the first prompt as inputs to a first generative AI, thereby obtaining emotion-based data (e.g., first data) by content within the group.

[0227] In operation 807, the processor (120) (e.g., the prompt generation unit (570) of FIG. 5) may generate a second prompt for generating a summary based on the first data and the sentiment keywords of all groups. In one embodiment, the processor (120) may generate the second prompt to generate a description of the searched contents (e.g., a sentiment-based second description (e.g., a summary) for the entire contents). In one embodiment, the processor (120) may also generate the prompt using a predefined template for the second prompt that is passed as an input to an AI model for generating a content-based description.

[0228] At step 809, the processor (120) may transmit the second prompt as input to the generative AI. In one embodiment, the processor (120) may provide the second prompt to the generative AI model of the on-device and / or the generative AI model of the server (e.g., an LLM or an Image Representation AI model). In one embodiment, the processor (120) may transmit the entire content and the second prompt as input to the generative AI model (e.g., an LLM or an Image Representation AI model, etc.) to generate an emotion-based second description (e.g., a summary) for the entire content.

[0229] In operation 811, the processor (120) may obtain group-specific emotion-based second data. According to one embodiment, the processor (120) may obtain the second data based on the entire content and the second prompt. According to one embodiment, the processor (120) may obtain second data including a second description from a generative AI model based on the second prompt. According to one embodiment, the processor (120) may obtain (or generate) second data according to a description generation process (e.g., a second description generation operation) executed in relation to the prompt in an on-device AI model. According to one embodiment, the processor (120) may obtain (or receive) second data according to a description generation process (e.g., a second description generation operation) executed in relation to the second prompt in a server AI model from a server.

[0230] According to one embodiment, the processor (120) may generate a second prompt for inferring the cause of an emotion related to an emotion keyword based on the group-specific content, and transmit the group-specific content and the second prompt as input to a second generative AI, thereby obtaining emotion-based data (e.g., second data) for the entire content.

[0231] In operation 813, the processor (120) (e.g., the interface generation and provision unit (580) of FIG. 5) may display representative content for each group and a second interface based on second data. According to one embodiment, the processor (120) may display a second interface (e.g., see FIG. 13) on the display (160) based on the representative content for each group and the second data (e.g., an emotion-based second description).

[0232] In operation 815, the processor (120) (e.g., the interface generation and provision unit (580) of FIG. 5 ) may receive an input for selecting a group based on the second interface. In one embodiment, the processor (120) may receive an input for selecting a group based on the second interface while displaying the second interface. For example, the processor (120) may receive an input for selecting a portion corresponding to one of the groups displayed via the second interface.

[0233] In operation 817, the processor (120) (e.g., the interface generation and provision unit (580) of FIG. 5) may display a first interface based on content and first data related to the selected group. According to one embodiment, in response to an input for selecting a group, the processor (120) may display a first interface (e.g., see FIG. 14) on the display (160) based on content and first data (e.g., an emotion-based first description) related to the selected group (e.g., switching a second interface to the first interface).

[0234] FIG. 9 is a diagram illustrating an example of an operation of searching for and providing emotion-based content in an electronic device according to one embodiment of the present disclosure.

[0235] As illustrated in FIG. 9, the electronic device (101) may receive a request from a user to search for content (e.g., images / videos) within the electronic device (101) based on emotions. For example, the electronic device (101) may receive an input of a search term (901) including an emotion-related keyword (e.g., "happiness") from the user (e.g., "Find me pictures that make me happy in the past 6 months").

[0236] According to one embodiment, the electronic device (101) may, in response to receiving a search request, search for content corresponding to a search term (901) in content (903) within the electronic device (101). In one embodiment, the electronic device (101) may search for content (e.g., images / videos) related to a corresponding emotion (e.g., emotions related to emotion keywords extracted from the search term (901)) using tags (e.g., user-defined tags, hidden tags at various points in time based on device analysis, etc.) stored and managed within the electronic device (101). According to one embodiment, the electronic device (101) may search for content related to a corresponding emotion according to an emotion keyword of the search term (901) based at least on health sensor data (910), image analysis (920), and / or user settings (930). For example, the electronic device (101) may include an operation to search for an image (e.g., a photograph) containing an emotional hidden tag based on health sensor data (910) and / or image analysis (920). For example, the electronic device (101) may include an operation to search for an image containing an emotional tag and / or an image containing a preference based on a user setting (930).

[0237] According to one embodiment, the electronic device (101) may generate emotional information about content (903) based on health sensor data (910) and store the generated emotional information in the metadata of the content. For example, the electronic device (101) may determine the emotional information of the user based on health sensor data acquired by the user's device (e.g., the electronic device (101), a wearable device worn by the user, etc.) at the time the user takes an image or video. In one embodiment, the health sensor data may include, for example, biometric data such as heart rate, pulse, and / or brain waves, and data detected by various sensors included in the wearable device, such as an inertial measurement unit (IMU) sensor (e.g., an acceleration sensor, an angular velocity sensor (or a gyroscope, and / or a magnetometer, etc.).

[0238] According to one embodiment, the electronic device (101) may determine the user's emotional information based on health sensor data and store the emotional information in metadata (e.g., tags, etc.) of the corresponding content (e.g., content (e.g., images and / or videos) captured at a similar time to the time of acquiring the health sensor data). For example, the electronic device (101) may determine the user's emotional information based on health sensor data acquired by the user's device (e.g., the electronic device (101), a wearable device worn by the user, etc.) at the time the user checks the content, shares the content, or edits the content, and store the emotional information in metadata of the corresponding content. According to one embodiment, when the electronic device (101) determines emotional information for the content based on health sensor data, the emotional information may be displayed on the screen of the electronic device (101) or managed as hidden data (e.g., hidden tags) that are not displayed on the screen of the electronic device (101).

[0239] According to one embodiment, the electronic device (101) can determine emotional information based on at least one of health sensor data, content information, and / or content metadata. According to one embodiment, an AI model trained to output emotional information based on at least one of health sensor data, content information, and / or content metadata can be used to determine emotional information based on health sensor data. According to one embodiment, emotional information related to content (e.g., tags, hidden tags, etc.) can be utilized when searching for content.

[0240] According to one embodiment, the electronic device (101) may generate emotional information about the content (903) based on image analysis (920) of the content. According to one embodiment, the electronic device (101) may generate emotional information based on at least some of scene analysis of the image, analysis of character characteristics (e.g., facial expression, posture, etc.) within the image, and / or analysis of object characteristics within the image. According to one embodiment, the electronic device (101) may generate emotional information by referencing user personalized information stored in the electronic device (101) and / or an external device (e.g., a server, a cloud, etc.) when analyzing the image. For example, the electronic device (101) may determine that an object preferred by the user (e.g., a flower, coffee, a gift box, etc.) is included in the image or video, and may estimate a positive emotion (e.g., joy, pleasure, happiness, etc.) to generate emotional information. For example, the electronic device (101) can generate emotional information by estimating positive emotions by detecting that the image or video is a user's preferred location (e.g., the sea) and / or weather (e.g., clear) based on scene analysis.

[0241] According to one embodiment, the electronic device (101) can generate emotional information about content using an AI model trained to output emotional information based on image analysis. According to one embodiment, the emotional information generated based on image / video information can be stored in metadata (e.g., tags, hidden tags, etc.) of the corresponding content or managed in association with the corresponding content in the electronic device (101). The emotional information can be stored in the form of tags or in a data format defined by the electronic device. According to one embodiment, the emotional information related to the content can be utilized when searching for the content.

[0242] According to one embodiment, the electronic device (101) may sort and group (905) the searched contents based on reference information (e.g., time, place, object, person, weather, etc.). In one embodiment, the example of FIG. 9 may illustrate that the reference information includes a place. For example, the electronic device (101) may create a first group (Group 1) related to place A, a second group (Group 2) related to place B, a third group (Group 3) related to place C, and a fourth group (Group 4) related to place D. According to one embodiment, the contents for each group may include contents in which an emotion tag (e.g., a user-defined tag and / or a device analysis-based hidden tag) corresponding to an emotion keyword is set among the contents (903) in the electronic device (101) based on the emotion keyword of the search word (901). According to one embodiment, the electronic device (101) can infer emotions based on emotion analysis of content (903) within the electronic device (101), and / or identify emotions based on emotion tags set in content (903) within the electronic device (101). According to one embodiment, the electronic device (101) can search for content related to emotion keywords based on inference and / or identification related to content (903).

[0243] According to one embodiment, the electronic device (101) may group searched contents (e.g., images and / or videos, etc.) based on predetermined criteria information (e.g., time, place, person, and / or object, etc.). According to one embodiment, if the number of searched contents in each group is less than or equal to a predetermined number or if it is determined that it is impossible to generate an emotion-based cause explanation (e.g., emotion information) using an AI model (e.g., an image representation AI model or a generative AI) using the content, the electronic device (101) may operate to additionally search and acquire similar contents based on criteria information in each group.

[0244] According to one embodiment, the electronic device (101) may group content based on predetermined reference information, including content that is primarily acquired (e.g., representative content (940, 950, 960, 970)) and content that is secondarily acquired through additional search of similar content (e.g., similar content (or sub-content (945, 955, 975)). For example, the content of the first group (Group 1), the second group (Group 2), the third group (Group 3), and the fourth group (Group 4) according to the grouping (905) may include, for example, primary search result data (e.g., metadata-based search data) (e.g., representative content (940, 950, 960, 970)) and / or secondary search result data (e.g., secondary search data (e.g., similar content) additionally searched to richly explain the cause of emotion) (e.g., sub-content (945, 955, 975)) may be included.

[0245] In one embodiment, the electronic device (101) may obtain and provide a description (e.g., emotional information) to the user explaining how the corresponding result data is related to the emotional keyword through the generative AI. In one embodiment, the electronic device (101) may generate a place-based name (907) for each group, an emotion-based description (909) for each group, and a description summary (911) for each group based on the generative AI, and may obtain (or receive) the description (e.g., emotional information or emotional data) generated through the generative AI.

[0246] According to one embodiment, the electronic device (101) may generate a name (907) of the corresponding place in association with a designated application (e.g., a map application) based on metadata information (e.g., location information or place information, etc.) of contents (e.g., representative contents (940, 950, 960, 970) and / or similar contents (or sub contents (945, 955, 975)) within each group. According to one embodiment, the electronic device (101) may also generate a name for the corresponding place by identifying an object of the contents within each group (e.g., identification through text OCR and / or object image identification through image analysis).

[0247] According to one embodiment, the electronic device (101) can generate a description (909) explaining the cause of the emotion based on the contents within the group. For example, the electronic device (101) can input a prompt to generate a description (909) explaining the cause of the emotion to a generative AI (e.g., an LLM or an Image representation AI model) and obtain the description (909) by generating it through the generative AI. For example, the electronic device (101) can infer the cause of the emotion by using the content and the emotion keyword to generate the description (909) (e.g., emotion information). For example, if the electronic device (101) identifies a 'trip to Paris' through scene analysis of an image and identifies 'happiness' as an emotion keyword, it can generate the description 'I am happy because of the pleasant trip to Paris'.

[0248] According to one embodiment, the electronic device (101) may generate a prompt to generate a description (911) of contents within a group (e.g., an emotion-based description (e.g., a summary (911)) for the entire contents. According to one embodiment, the electronic device (101) may also generate a prompt using a predefined template for the prompt that is passed as input to an AI model for generating a content-based description (e.g., a summary (911)). According to one embodiment, the electronic device (101) may pass the prompt as input to a generative AI model (e.g., an LLM or an Image Representation AI model). According to one embodiment, the electronic device (101) may pass the entire contents and the prompt as input to a generative AI model (e.g., an LLM or an Image Representation AI model, etc.) to generate an emotion-based description (e.g., a summary (911)) for the entire contents.

[0249] In one embodiment, the electronic device (101) can provide a user with a description generated through generative AI, associated with content by group. For example, the electronic device (101) can display emotional keyword-based search results (913) through a designated interface.

[0250] FIG. 10 is a diagram illustrating an example of an operation of searching for and providing emotion-based content in an electronic device according to one embodiment of the present disclosure.

[0251] Referring to FIG. 10, FIG. 10 may represent an example of searching for content within an electronic device (101) (e.g., searching through a content search engine (530)) based on a user search request that includes emotion, using emotion tags at various points in time and user-configured data (e.g., 'like' preference, etc.), and classifying and sorting the searched content (e.g., representative content (940, 950, 960, 970) and similar content (945, 955, 975) of FIG. 9) by reference information (e.g., time and place) and providing the same.

[0252] According to one embodiment, the electronic device (101) can perform a content search (e.g., primary search) within the electronic device (101). According to one embodiment, the electronic device (101) can search for content (1010) (e.g., representative content (940, 950, 960, 970) of FIG. 9) (e.g., images and / or videos) in which an emotion tag (e.g., a user-defined tag and / or a device analysis-based hidden tag) corresponding to the emotion keyword is set among the content within the electronic device (101) based on the emotion keyword.

[0253] According to one embodiment, the electronic device (101) may group searched content (1010) (e.g., primary searched content) based on predetermined reference information (e.g., time, location, person, and / or object, etc.), and classify and sort each group. In one embodiment, FIG. 10 may illustrate an example in which the reference information includes time information and location information, and the searched content (1010) is grouped and provided based on time and location.

[0254] According to one embodiment, the electronic device (101) may create a first group (Group 1) (e.g., January 24 group), a second group (Group 2) (e.g., February 24 group), a third group (Group 3) (e.g., April 24 group), and a fourth group (Group 4) (e.g., May 24 group) based on reference information (e.g., time and place) of the searched content (1010). According to one embodiment, the content for each group (e.g., representative contents (940, 950, 960, 970) of FIG. 9) may include content in which an emotion tag (e.g., a user-set based tag and / or a device analysis based hidden tag) corresponding to an emotion keyword is set among the contents in the electronic device (101) based on the emotion keyword of the search word.

[0255] According to one embodiment, the electronic device (101) can determine whether the searched content (1010) (e.g., content within each group according to the first search) has a predetermined condition. For example, the electronic device (101) can determine whether the searched content (1010) is less than a predefined number (e.g., about 2 pages), or whether it is difficult to explain the cause of the emotion based on the result of analysis (e.g., image analysis) of the searched content (1010), or whether it is insufficient as an input value for an AI model (e.g., an image representation AI model, a multimodal LLM, etc.) to be used for generating an explanation of the cause of the emotion.

[0256] According to one embodiment, if the electronic device (101) determines that the number of searched contents (1010) in each group is less than or equal to a predetermined number or that it is impossible to generate an emotion-based cause explanation (e.g., emotion information) using an AI model (e.g., an image representation AI model or a generative AI) using the searched contents (1010), the electronic device (101) may perform an additional search (e.g., a secondary search) for similar contents based on reference information in each group.

[0257] According to one embodiment, if the electronic device (101) determines that the initially searched content (1010) (e.g., representative content (940, 950, 960, 970)) does not have a predetermined condition, the electronic device (101) may additionally search for content to be included in the corresponding group that does not have the predetermined condition. According to one embodiment, the electronic device (101) may perform an additional search for similar content (1020) (e.g., sub-contents (945, 955, 975) of FIG. 9) that includes information (e.g., time information, location information, location information) related to a similar time and / or location as the initially searched content (1010). For example, the electronic device (101) may perform an operation of additionally searching and acquiring similar content (1020) that includes a wider variety of objects.

[0258] According to one embodiment, the electronic device (101) may group content based on predetermined criteria information, including content (1010) searched primarily and content (1020) searched secondary through additional searches (e.g., similar content). For example, the electronic device (101) may group similar content (1020) searched secondary into each group grouped based on content (1010) searched primary. For example, the contents of the first group (Group 1), the second group (Group 2), the third group (Group 3), and the fourth group (Group 4) may include, for example, contents (1010) according to the first search (e.g., representative contents (940, 950, 960, 970) of FIG. 9) and / or similar contents (1020) according to the second search (e.g., contents additionally searched to richly explain the cause of the emotion, for example, ㅇ슈아 contents (945, 955, 975) of FIG. 9).

[0259] FIG. 11 is a diagram illustrating an example of an operation of generating and providing emotional information related to search content in an electronic device according to one embodiment of the present disclosure.

[0260] Referring to FIG. 11, FIG. 11 can show a schematic example of the input / output process of an AI model that generates a description of the cause of an emotion based on searched content.

[0261] According to one embodiment, example (A) may represent an example of an operation of generating an emotion-based description (e.g., a first description) for each group using a generative AI model.

[0262] According to one embodiment, the operation of example (A) may represent an example of an operation of passing a prompt (1120) (e.g., a text prompt) for generating content (1110) and an emotion keyword-based description (1140) of each group as input to an Image Representation AI model (1130), and generating and outputting a description (1140) of the cause of the emotion based on content analysis (e.g., image analysis) and depicted content (e.g., scene analysis, object image-based, etc.) for each group.

[0263] According to one embodiment, in the operation of example (A), when the electronic device (101) (e.g., the prompt generation unit (570) of FIG. 5) generates the prompt (1120), the electronic device (101) may generate the prompt (1120) that is passed as an input to an image representation AI model (1130) (e.g., an image to text conversion AI model, a multimodal LLM, etc.) for generating a description (1140) of the content (1110). According to one embodiment, when the electronic device (101) generates the prompt (1120), the electronic device (101) may also generate the prompt (1120) by using a predefined template for the prompt (1120). For example, in order to utilize 'extracted emotion keywords' from a search request entered by a user, a data field that can include keywords for emotions can be defined, and a prompt (1120) can be generated that instructs 'to generate a description that infers the cause of the emotion through descriptions of contents input to the Image representation AI model (1130)'. According to one embodiment, the electronic device (101) can transmit the generated prompt (1120) as an input to the Image representation AI model (1130).

[0264] According to one embodiment, the electronic device (101) (e.g., the prompt generation unit (570) of FIG. 5) may generate a first prompt (1120) for inferring the cause of an emotion related to an emotion keyword based on group-specific content (1110). According to one embodiment, the electronic device (101) may generate a first prompt (1120) for generating a description (1140) of the searched contents (e.g., a first description based on emotion for each group). According to one embodiment, the electronic device (101) may transmit the first prompt (1120) as an input to an AI model (e.g., an LLM or an Image representation AI model, etc.) for generating a content-based description (1140). According to one embodiment, the electronic device (101) may transmit group-specific content (1110) and a first prompt (1120) as inputs to an AI model (e.g., an LLM or Image Representation AI model, etc.) to generate an emotion-based first description (1140) for each group.

[0265] According to one embodiment, the Image representation AI model (1130) can generate a description (1140) explaining the cause of the emotion based on the contents within the group and the emotion. For example, the Image representation AI model (1130) can infer the cause of the emotion through the content (1110) and the emotion keyword and generate a description (1140) (e.g., emotion information). For example, if a 'Paris trip' is identified through scene analysis of an image and 'happiness' is identified as the emotion keyword, the description (1140) 'I am happy because of the pleasant trip to Paris' can be generated.

[0266] According to one embodiment, based on the operation of example (A), the generated description (1140) (e.g., the first description) may be displayed on a display in response to a user input indicating an intent to view details of each group through a designated interface (e.g., see the first interface of FIG. 14).

[0267] In one embodiment, example (B) may illustrate an example of an operation that uses a generative AI model to generate an emotion-based descriptor (e.g., a second descriptor (or summary)) for each group.

[0268] According to one embodiment, the operation of example (B) may represent an operation of generating a single summarized description (1170) (e.g., a second description (1170) (or summary)) based on the entire description (1140) (e.g., a first description (1140)) of the cause of each group of emotions generated by the image representation AI model (1130). Based on the operation of example (B), the generated description (1170) may be used as content that summarizes and explains the cause of the emotion requested by the user based on the entire searched content in the first screen (e.g., a second interface) that displays the results of a user search request.

[0269] According to one embodiment, in the operation of example (B), the electronic device (101) (e.g., the prompt generation unit (570) of FIG. 5) may generate a second prompt (1150) for generating a summary based on an emotional keyword, including a first description (1140) for all groups of content (1110). According to one embodiment, the electronic device (101) may generate a second prompt (1150) for generating an emotional-based second description (1170) (e.g., a summary) for the entire content (1110). According to one embodiment, the electronic device (101) may pass the second prompt (1150) as an input to an AI model (1160) (e.g., an AI model for summarization) (e.g., an LLM or an Image Representation AI model, etc.) for generating a content-based description (e.g., a summary), and may generate the second prompt (1150) using a predefined template for the second prompt (1150). In one embodiment, the electronic device (101) may pass the entire content (1110) and the second prompt (1150) as inputs to an AI model (e.g., an LLM or Image Representation AI model, etc.) to generate an emotion-based second description (1150) (e.g., a summary) for the entire content (1110).

[0270] According to one embodiment, based on the operation of example (B), the generated description (1170) (e.g., the second description) may be displayed on a display in response to a user input indicating an intent to view a summary of the entire content through a designated interface (e.g., see the second interface of FIG. 13).

[0271] FIG. 12 is a diagram illustrating an example of an operation of providing content group-specific descriptions in an electronic device according to one embodiment of the present disclosure.

[0272] Referring to FIG. 12, FIG. 12 can generate and provide an explanatory text (1220) (e.g., see the first explanatory text (1140) of FIG. 11) explaining the cause of an emotion through a generative AI (e.g., Image representation AI) using the final content configured for each group.

[0273] According to one embodiment, the electronic device (101) may classify and group contents (1210) based on emotion tags (e.g., user-defined tags and / or device analysis-based hidden tags) related to the same emotion and / or similar emotions. According to one embodiment, the electronic device (101) may obtain descriptions (1220) explaining the cause of emotion for each group from a generative AI, and provide the descriptions (1220) by mapping them for each group. For example, a description explaining the cause of emotion for contents of the first group may be mapped to a first group (Group 1), a description explaining the cause of emotion for contents of the second group may be mapped to a second group (Group 2), a description explaining the cause of emotion for contents of the third group may be mapped to a third group (Group 3), and a description explaining the cause of emotion for contents of the fourth group may be mapped to a fourth group (Group 4).

[0274] FIG. 13 is a diagram illustrating an example of an interface provided by an electronic device according to one embodiment of the present disclosure.

[0275] According to one embodiment, FIG. 13 may illustrate an example of an interface (e.g., a second interface) that displays each group of the entire searched content sorted by reference information (e.g., timeline and location).

[0276] In one embodiment, the interface may include a first area (1310) for group-specific content (e.g., a result screen display portion), a second area (1320) for a description (e.g., a second description) summarizing the cause of the sentiment of the entire group, and a third area (1330) for a title or user search term. In one embodiment, the interface may be displayed as an interface in which the first area (1310), the second area (1320), and the third area (1330) are connected in a hierarchical structure.

[0277] In one embodiment, the interface includes a timeline-style interface, and can display each group by sorting it based on its chronological characteristics. In one embodiment, the interface can include a representative emotional image (1300) (or emotional indicator) regarding the entire content.

[0278] According to one embodiment, the electronic device (101) can identify a representative emotion related to the topic of the entire content based on data acquired from the AI ​​model. According to one embodiment, the electronic device (101) can generate an emotional image (1300) (or an emotional indicator) representing the identified representative emotion. According to one embodiment, the electronic device (101) can display the emotional image (1300) by arranging it along with content by group based on the first region (1310).

[0279] FIG. 14 is a diagram illustrating an example of an interface provided by an electronic device according to one embodiment of the present disclosure.

[0280] According to one embodiment, FIG. 14 may illustrate an example of an interface (e.g., a first interface) that is displayed when a user selects a UI graphical element representing a content group (e.g., a market square group) in the interface (e.g., a second interface) of FIG. 13.

[0281] According to one embodiment, the interface may be displayed as an interface hierarchically connected with a first area (1410) for content within a group (e.g., a portion displaying content of the group), a second area (1420) for a description (e.g., a first description) explaining the cause of the emotion based on the content within the group, and a third area (1430) for a title or user search term (e.g., corresponding to the third area (1330) of FIG. 13).

[0282] In one embodiment, the interface (e.g., the first interface and / or the first interface) may include a representative emotional image regarding the entire content based on data. In one embodiment, the processor (120) may identify a representative emotional image regarding the topic of the entire content based on the data. In one embodiment, the processor (120) may generate an emotional image representing the representative emotional image. In one embodiment, the processor (120) may place the emotional image on the interface and display it along with the content.

[0283] According to one embodiment, the electronic device (101) may display a second interface (see the second interface of FIG. 13) including representative content and an emotion-based second description for each group on the display (160). According to one embodiment, while displaying the second interface, the electronic device (101) may receive an input for selecting a portion of a group based on the second interface. For example, the electronic device (101) may receive an input for selecting a portion corresponding to one of the groups displayed through the second interface.

[0284] According to one embodiment, in response to an input for selecting a group, the electronic device (101) may display a first interface (e.g., refer to the first interface of FIG. 14) on the display (160) that includes content related to the selected group (e.g., content within the selected group) and an emotion-based first description. For example, the electronic device (101) may switch the second interface of FIG. 13 to the first interface of FIG. 14 and display it.

[0285] According to one embodiment, as illustrated in FIGS. 13 and 14 , the electronic device (101) may provide results in response to a user's search request by arranging each grouped content in a timeline format. For example, the electronic device (101) may display an emotion-based description of all searched and grouped content in a summary format in an interface (e.g., a second interface) that displays the results.

[0286] For example, when a user selects a graphic element (e.g., a content object or an image object) representing each group in the second interface, the electronic device (101) may display and provide a description explaining the reason based on emotion only for the contents included in the corresponding group. For example, the electronic device (101) may switch from the second interface to the first interface and display a description explaining the reason based on emotion only for the contents included in the group selected by the user.

[0287] FIG. 15 is a flowchart illustrating an operation method of an electronic device according to one embodiment of the present disclosure.

[0288] According to one embodiment, FIG. 15 may illustrate an example of a method for generating emotional tags for content (e.g., images and / or videos) and setting content groups when acquiring content from an electronic device (101) according to one embodiment. For example, FIG. 15 may illustrate examples of various emotional tags that may be generated for content when acquiring the content and / or analyzing the content.

[0289] An operation method performed in an electronic device (101) according to one embodiment of the present disclosure may be performed, for example, according to a flowchart illustrated in FIG. 15. The flowchart illustrated in FIG. 15 is an example according to one embodiment of an operation of the electronic device (101), and the order of at least some operations may be changed, performed in parallel, performed as independent operations, or at least some other operations may be performed complementarily to at least some operations. According to one embodiment of the present disclosure, operations 1501 to 1511 may be performed in at least one processor (120) of the electronic device (101).

[0290] As illustrated in FIG. 15, an operation method performed by an electronic device (101) according to one embodiment may include an operation of obtaining content (operation 1501), an operation of identifying an emotion (operation 1503), an operation of setting an emotion tag (operation 1505), an operation of determining whether a similar emotion content group previously created exists (operation 1507), an operation of updating the similar emotion content group previously created when there is a similar emotion content group previously created (operation 1509), and an operation of storing content and / or creating a group when there is no similar emotion content group previously created (operation 1511).

[0291] Referring to FIG. 15, in operation 1501, the processor (120) of the electronic device (101) may obtain content (e.g., images and / or videos). According to one embodiment, the processor (120) may obtain content based on taking pictures using a camera and / or receiving (e.g., downloading) the content from an external device (e.g., a content server and / or another electronic device). In operation 1503, the processor (120) may infer emotions related to the content based on the content acquisition. For example, the processor (120) may infer emotions at the time of taking pictures of the content, emotions at the time of analyzing the image, emotions at the time of checking the image, and / or emotions at the time of user setting.

[0292] According to one embodiment, the processor (120) may infer emotions using health sensor data (e.g., the user's biometric information) acquired through sensors capable of detecting various health information included in the user's wearable device. For example, the processor (120) may determine the user's emotional information based on health sensor data acquired by the user's device (e.g., the electronic device (101) and / or the wearable device worn by the user) at the time the user captures content.

[0293] In one embodiment, the processor (120) may determine emotional information about the content based on image analysis of the content. For example, the processor (120) may infer emotional information based on at least some of the following: scene analysis of the content, analysis of character characteristics (e.g., facial expressions and / or postures) included in the content, and / or analysis of object characteristics within the content.

[0294] According to one embodiment, the processor (120) may generate emotional information by referencing user personalized information stored in the electronic device (101) and / or an external device (e.g., a server, a cloud, etc.) during image analysis. For example, the processor (120) may determine that an image or video contains an object preferred by the user (e.g., a flower, coffee, a gift box, etc.) and estimate a positive emotion (e.g., joy, pleasure, happiness, etc.) to generate emotional information. For example, the processor (120) may detect that an image or video contains a location (e.g., the sea) and / or weather (e.g., clear) preferred by the user based on scene analysis and estimate a positive emotion to generate emotional information. According to one embodiment, the processor (120) may also generate emotional information about content by using an AI model trained to output emotional information based on image analysis.

[0295] In operation 1505, the processor (120) may set an emotion tag. According to one embodiment, the processor (120) may infer an emotion based on a time point and / or situation, and set an emotion tag of the inferred emotion to the acquired content. According to one embodiment, even if the user does not directly set a tag to the content, the processor (120) may internally determine (or infer) an emotion and generate a hidden emotion tag that is not visible to the user. According to one embodiment, the processor (120) may store and manage the time point and cause information of the emotion tag generation together in the memory (130) (e.g., DB (540) of FIG. 5).

[0296] In operation 1507, the processor (120) may determine whether a group of similar emotional content already generated exists. According to one embodiment, when new content (e.g., images and / or videos, etc.) is acquired, the processor (120) may generate an emotional tag through the emotional inference process of operations 1503 and 1505, and update a group of content in relation to the emotional tag.

[0297] In operation 1507, if a similar emotional content group already created exists (e.g., 'yes' in operation 1507), the processor (120) may update the content group in operation 1509. For example, the processor (120) may perform an operation of adding the acquired content to the content group and generating a description.

[0298] In operation 1507, if there is no previously generated similar emotional content group (e.g., 'No' in operation 1507), in operation 1511, the processor (120) may store the acquired content and / or generate a new emotional content group based on the acquired content and the emotional tag of the content.

[0299] An operation method performed in an electronic device (101) according to one embodiment of the present disclosure may include an operation of receiving an input of a search word requesting a content search related to an emotion. The operation method may include an operation of extracting an emotion keyword from the search word in response to the input. The operation method may include an operation of performing a content search within the electronic device based on the emotion keyword. The operation method may include an operation of grouping the searched content based on time and location. The operation method may include an operation of generating a prompt for inferring the cause of the emotion related to the emotion keyword based on the content for each group. The operation method may include an operation of transmitting the content for each group and the prompt to a generative artificial intelligence (AI) of an on-device and / or a server. The operation method may include an operation of acquiring data based on the content for each group and the prompt. The operation method may include an operation of displaying an interface including the content for each group and the data on a display based on the data.

[0300] According to one embodiment, the operation of performing the search may include an operation of determining whether the searched content satisfies a set condition, and if the set condition is not satisfied, an operation of searching for similar content by group based on time and / or location related to content within the group that does not satisfy the set condition.

[0301] In one embodiment, the act of generating the prompt may include an act of generating the prompt based on the retrieved content having a defined condition.

[0302] According to one embodiment, the operation of obtaining the data may include an operation of generating a first prompt for inferring a cause of an emotion related to an emotion keyword based on content by group, an operation of transmitting the content by group and the first prompt as inputs to a first generative AI, and an operation of obtaining emotion-based first data by content within the group. According to one embodiment, the operation of obtaining the data may include an operation of generating a second prompt for generating a summary based on the first data and emotion keywords of all groups, an operation of transmitting the second prompt as inputs to a second generative AI, and an operation of obtaining emotion-based second data by group.

[0303] In one embodiment, the first data may be obtained from the first generative AI and may include an emotion-based description of the content within each group. In one embodiment, the second data may be obtained from the second generative AI and may include an emotion-based description of the summary of each group.

[0304] According to one embodiment, the operation of displaying the interface may include an operation of displaying a second interface based on representative content and second data for each group, an operation of receiving an input for selecting a group based on the second interface, and an operation of displaying a first interface based on content and first data related to the selected group in response to the input.

[0305] According to one embodiment, the act of displaying the interface may include an act of displaying each group by arranging it based on a timeline interface.

[0306] In one embodiment, the interface may include a representative emotional image for the entire content based on the data.

[0307] According to one embodiment, the method may include an operation of inferring an emotion based on an emotion analysis of content within the electronic device and / or an operation of identifying an emotion based on an emotion tag set in content within the electronic device, and an operation of searching for content related to the emotion keyword based on the inference and / or identification related to the content.

[0308] A non-transitory computer-readable recording medium storing instructions that, when executed by a processor (120) of an electronic device (101) according to one embodiment of the present disclosure, cause the processor (120) to perform operations, wherein the instructions, when executed by the processor, cause the electronic device to receive an input of a search word requesting a search for content related to an emotion, extract an emotion keyword from the search word in response to the input, perform a content search within the electronic device based on the emotion keyword, group the searched content based on time and place, generate a prompt for inferring a cause of an emotion related to the emotion keyword based on the content by group, transmit the content by group and the prompt to a generative artificial intelligence (AI) of an on-device and / or a server, acquire data based on the content by group and the prompt, and generate an interface including the content by group and the data based on the data. It may include a recording medium that causes the display to perform an action.

[0309] It will be appreciated that the above-described embodiments and their technical features may be combined with each other in any and all combinations, as long as there is no potential conflict between the two embodiments or features. For example, any and all combinations of two or more of the above-described embodiments may be envisioned and incorporated within the present disclosure. One or more features from any embodiment may be incorporated into any other embodiment, providing a corresponding advantage or advantages.

[0310] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

[0311] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another component (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0312] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0313] Various embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more commands stored in a storage medium (or recording medium) (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one command among the one or more commands stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one command called. The one or more commands may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g. electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0314] According to one embodiment, the method according to various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play Store™) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium (or recording medium), such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0315] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and arranged in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0316] The various embodiments of the present disclosure disclosed in this specification and drawings are intended to provide specific examples to facilitate easy explanation of the technical content of the present disclosure and to aid understanding of the present disclosure, and are not intended to limit the scope of the present disclosure. Therefore, the scope of the present disclosure should be interpreted to include all modifications or variations derived based on the technical concepts of the present disclosure, in addition to the embodiments disclosed herein.

Claims

1. In electronic devices (101, 201, 400), display(160, 440); At least one processor (120, 230, 410) comprising processing circuitry; and Contains a memory (130, 240, 420) for storing instructions, The above instructions, when executed by the at least one processor, cause the electronic device to: Receive input of search terms requesting search for content related to emotions, In response to the above input, extract emotional keywords from the search term, Perform content search within electronic devices based on the above emotional keywords, Group searched content based on time and location, Generate a prompt to infer the cause of the emotion related to the above emotion keyword based on the content of each group, The above group-specific content and the above prompt are delivered to the on-device and / or server-generated artificial intelligence (AI), Obtain data based on the above group-specific content and the above prompt, and An electronic device that displays an interface including group-specific content and data on a display based on the above data.

2. In the first paragraph, when the instructions are executed by the at least one processor, the electronic device, Determine whether the searched content satisfies the specified conditions, An electronic device that searches for similar content by group based on time and / or location related to content within a group that does not satisfy the above-described conditions, if the above-described conditions are not satisfied.

3. In the first paragraph, when the instructions are executed by the at least one processor, the electronic device, An electronic device that generates the prompt based on whether the searched content has a defined condition.

4. In the first paragraph, when the instructions are executed by the at least one processor, the electronic device, Generate a first prompt to infer the cause of the emotion related to the emotion keyword based on the content of each group, The group content and the first prompt are passed as input to the first generative AI, Obtain the first data based on emotions for each content within the group, Create a second prompt to generate a summary based on the first data and sentiment keywords of all groups, Passing the above second prompt as input to the second generative AI, and An electronic device that obtains group-specific emotion-based secondary data.

5. In paragraph 4, The first data is obtained from the first generative AI and includes an emotion-based description of the content within each group, An electronic device wherein the second data is obtained from the second generative AI and includes an emotion-based description of the summary of each group.

6. In paragraph 4, An electronic device wherein the first generative AI and the second generative AI are one generative AI or include separate generative AIs.

7. In the fourth paragraph, when the instructions are executed by the at least one processor, the electronic device, An electronic device that displays a second interface based on representative content and second data for each group.

8. In the 7th paragraph, when the instructions are executed by the at least one processor, the electronic device, Receive an input for selecting a group based on the second interface, An electronic device, responsive to the above input, for displaying a first interface based on content and first data related to the selected group.

9. In the first paragraph, when the instructions are executed by the at least one processor, the electronic device, An electronic device that displays each group in an aligned manner based on a timeline interface.

10. In paragraph 1, The above interface is an electronic device that includes a representative emotional image regarding the entire content based on the above data.

11. In the first paragraph, when the instructions are executed by the at least one processor, the electronic device, Based on the above data, identify representative sentiments regarding the topic of the entire content, Generate an emotional image representing the above representative emotion, An electronic device that displays the above emotional image by placing it on the above interface.

12. In the first paragraph, when the instructions are executed by the at least one processor, the electronic device, Inferring emotions based on sentiment analysis of content within the electronic device, and / or identifying emotions based on sentiment tags set on content within the electronic device; An electronic device that searches for content related to said emotional keyword based on inference and / or identification related to the content.

13. In the operating method of an electronic device (101, 200), An action that receives input of a search term requesting a search for content related to emotions; In response to the above input, an operation of extracting emotional keywords from the search word; An action to search for content within an electronic device based on the above emotional keywords; The act of grouping searched content based on time and location; An action to generate a prompt to infer the cause of the emotion related to the above emotion keyword based on the content of each group; An action to deliver the above group-specific content and the above prompt to the on-device and / or server-generated artificial intelligence (AI); An operation of obtaining data based on the above group-specific content and the above prompt; and A method comprising an action of displaying an interface including content and data for each group on the basis of the above data.

14. In the 13th paragraph, the operation of performing the search is as follows: An action that determines whether the searched content satisfies a set of conditions. A method comprising an operation of searching for similar content by group based on time and / or location related to content within a group that does not satisfy the above-described condition, if the above-described condition is not satisfied.

15. A non-transitory computer-readable medium storing instructions that, when executed by a processor of an electronic device, cause the processor to perform operations, The above instructions, when executed by the processor, cause the electronic device to: An action that receives input of a search term requesting a search for content related to emotions; In response to the above input, an operation of extracting emotional keywords from the search word; An action to search for content within an electronic device based on the above emotional keywords; The action of grouping searched content based on time and location, An action to generate a prompt to infer the cause of the emotion related to the above emotion keyword based on the content of each group. The action of delivering the above group-specific content and the above prompt to the on-device and / or server-generated artificial intelligence (AI), An operation of obtaining data based on the above group-specific content and the above prompt, and A recording medium that performs an operation of displaying an interface including the group-specific content and data on a display based on the above data.

Images