Electronic device, method, and non-transitory computer-readable storage medium for supporting associative learning

Abstract

The present disclosure provides a method of an electronic device. The method of the electronic device may comprise the operations of: generating correct answer information for information corresponding to a first user input; generating first hint information associated with the correct answer information by using a first generative AI model on the basis of the correct answer information and information related to a behavior pattern of a user, and providing the first hint information; obtaining a second user input including a first answer to the first hint information; evaluating the first answer on the basis of the correct answer information; and generating second hint information associated with the correct answer information by using the first generative AI model on the basis of the correct answer information, the information related to the behavior pattern, and first feedback information according to a result of the evaluation, and providing the second hint information. The first feedback information may include at least one of the first answer and the first hint information.

Description

Electronic devices, methods, and non-transitory computer-readable storage media supporting associative learning

[0001] The present disclosure relates to electronic devices, methods, and non-transitory computer-readable storage media that support associative learning.

[0002] Users experiencing memory or cognitive decline often struggle to accurately recall information they want to remember. Advances in technology have led to the development of various electronic devices (e.g., conversational robots) to assist these users. These devices monitor the user, provide contextual alerts or general information, and facilitate social interaction through interactions (e.g., simple conversations).

[0003] However, these electronic devices only provide simple information, guidance, and interaction based on established algorithms. They do not support services like associative learning, which enhance or alleviate the user's cognitive abilities. Associative learning refers to the process of learning to associate one stimulus or event with another. This associative learning can help strengthen or restore the user's memory by encouraging the user to actively recall the information they wish to remember.

[0004] The above information may be provided as background information to aid in understanding this document. None of the above is claimed to be prior art related to this document or can be used to determine prior art.

[0005] According to one embodiment, the electronic device may support associative learning.

[0006] According to one embodiment, an electronic device may be provided. The electronic device may include a display; a communication circuit; at least one processor including a processing circuit; and a memory including at least one storage medium storing instructions, wherein the instructions, when individually or collectively executed by the at least one processor, cause the electronic device to perform at least one operation. The at least one operation may include an operation of generating correct answer information for information corresponding to a first user input. The at least one operation may include an operation of generating first hint information associated with the correct answer information using a first generative artificial intelligence (AI) model based on the correct answer information and information related to a user's behavior pattern, and providing the first hint information. The at least one operation may include an operation of obtaining a second user input including a first answer to the first hint information. The at least one operation may include an operation of evaluating the first response based on the correct answer information. The at least one action may include an action of generating second hint information associated with the correct answer information using the first generative AI model based on the correct answer information, information related to the behavioral pattern, and first feedback information, based on the results of the evaluation, and providing the second hint information. The first feedback information may include at least one of the first response or the first hint information.

[0007] According to one embodiment, a method of operating an electronic device may be provided. The method of operating the electronic device may include at least one operation. The at least one operation may include an operation of generating correct answer information for information corresponding to a first user input. The at least one operation may include an operation of generating first hint information associated with the correct answer information using a first generative artificial intelligence (AI) model based on the correct answer information and information related to a user's behavioral pattern, and providing the first hint information. The at least one operation may include an operation of obtaining a second user input including a first response to the first hint information. The at least one operation may include an operation of evaluating the first response based on the correct answer information. The at least one operation may include an operation of generating second hint information associated with the correct information using the first generative AI model based on the correct answer information, information related to the behavioral pattern, and first feedback information, and providing the second hint information, based on a result of the evaluation. The first feedback information may include at least one of the first response or the first hint information.

[0008] According to one embodiment, a non-transitory computer-readable storage medium storing instructions may be provided. The instructions, when executed by at least a portion of at least one processor of an electronic device, may cause the electronic device to perform at least one operation. The at least one operation may include generating correct answer information for information corresponding to a first user input. The at least one operation may include generating first hint information associated with the correct answer information using a first generative artificial intelligence (AI) model based on the correct answer information and information related to a user's behavior pattern, and providing the first hint information. The at least one operation may include obtaining a second user input including a first answer to the first hint information. The at least one operation may include evaluating the first response based on the correct answer information. The at least one action may include an action of generating second hint information associated with the correct answer information using the first generative AI model based on the correct answer information, information related to the behavioral pattern, and first feedback information, based on the results of the evaluation, and providing the second hint information. The first feedback information may include at least one of the first response or the first hint information.

[0009] In connection with the description of the drawings, the same or similar reference numerals may be used for the same or similar components.

[0010] FIG. 1A is a block diagram of an electronic device within a network environment according to various embodiments of the present disclosure.

[0011] FIG. 1b is a diagram illustrating a generative artificial intelligence system according to one embodiment of the present disclosure.

[0012] FIG. 2 is a diagram illustrating an operation of monitoring a user and acquiring sensing data according to one embodiment of the present disclosure.

[0013] FIG. 3A is a diagram illustrating a mobile electronic device according to one embodiment of the present disclosure.

[0014] FIG. 3b is a block diagram of the mobile electronic device of FIG. 3a, according to one embodiment of the present disclosure.

[0015] FIG. 4A is a diagram illustrating a front side of a wearable electronic device according to one embodiment of the present disclosure.

[0016] FIG. 4B is a drawing showing the back of the wearable device of FIG. 4A according to one embodiment of the present disclosure.

[0017] FIG. 4C is a block diagram of the wearable device of FIG. 4A, according to one embodiment of the present disclosure.

[0018] FIG. 5 is a flowchart illustrating a method by which an electronic device supports associative learning according to one embodiment of the present disclosure.

[0019] FIG. 6 is a diagram illustrating a configuration of an electronic device for supporting associative learning according to one embodiment of the present disclosure.

[0020] FIG. 7 is a diagram for explaining the configuration of a multimodal model according to one embodiment of the present disclosure.

[0021] FIG. 8 is a flowchart illustrating a method by which an electronic device generates scene data and information related to a user's behavioral pattern, according to one embodiment of the present disclosure.

[0022] FIG. 9A is a diagram illustrating scene data according to one embodiment of the present disclosure.

[0023] FIG. 9b is a diagram illustrating information related to a user's behavior pattern according to one embodiment of the present disclosure.

[0024] FIG. 10 is a flowchart illustrating a method for an electronic device to generate scene data according to one embodiment of the present disclosure.

[0025] FIG. 11 is a flowchart illustrating a method for an electronic device to generate information related to a user's behavioral pattern, according to one embodiment of the present disclosure.

[0026] FIG. 12A is a diagram illustrating information that a user wants to remember, according to one embodiment of the present disclosure.

[0027] FIG. 12b is a diagram illustrating hint information associated with correct answer information corresponding to information that the user of FIG. 12a wants to remember, according to one embodiment of the present disclosure.

[0028] FIG. 13A is a diagram illustrating information that a user wants to remember, according to one embodiment of the present disclosure.

[0029] FIG. 13b is a diagram illustrating hint information associated with correct answer information corresponding to information that the user of FIG. 13a wants to remember, according to one embodiment of the present disclosure.

[0030] FIG. 14 is a flowchart illustrating a method for an electronic device to provide associative learning according to one embodiment of the present disclosure.

[0031] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. In connection with the description of the drawings, the same or similar reference numerals may be used for identical or similar components. Furthermore, in the drawings and related descriptions, descriptions of well-known functions and configurations may be omitted for clarity and conciseness.

[0032] FIG. 1A is a block diagram of an electronic device within a network environment according to various embodiments of the present disclosure.

[0033] Referring to FIG. 1A, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with an electronic device (104) or a server (108) via a second network (199) (e.g., a long-range wireless communication network). In one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0034] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or operations. According to one embodiment, as at least a part of the data processing or operations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or an auxiliary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together with the main processor (121). For example, when the electronic device (101) includes the main processor (121) and the auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a given function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0035] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, in the electronic device (101) itself where artificial intelligence is performed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0036] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0037] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0038] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0039] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0040] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0041] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0042] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0043] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0044] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0045] The haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. According to one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0046] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0047] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0048] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0049] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a communication module (192) (e.g., a cellular communication module, a short-range communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0050] The communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The communication module (192) may support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The communication module (192) may support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the communication module (192) may support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0051] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas, for example, by the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device via the at least one selected antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0052] According to various embodiments, the antenna module (197) may form a mmWave antenna module. In one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

[0053] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface).

[0054] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In one embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0055] The number of processors (120) may be one or more. For example, the processor (120) may have a multi-core processor structure such as a dual core, quad core, or hexa core.

[0056] The processor (120) can control the operations of the electronic device (101) by executing instructions stored in the memory (130). For example, the processor (120) can correspond to a plurality of processors that collectively perform a plurality of operations by dividing them among the processors.

[0057] FIG. 1b is a diagram illustrating a generative artificial intelligence system according to one embodiment of the present disclosure.

[0058] According to one embodiment, a user query / response interface (210) may receive input (e.g., user input or data acquired or generated by an electronic device (e.g., electronic device (101) of FIG. 1A). The data acquired or generated by the electronic device may include, for example, image or video data generated using a processor (e.g., processor (120) of FIG. 1A), values ​​transmitted via a sensor (e.g., sensor module (176) of FIG. 1A) or sensor hub (e.g., external illuminance, angle of the electronic device, temperature of the display (e.g., display module (160) of FIG. 1A) or electronic device, display size or expansion / reduction information, captured images of an image sensor). The user input may be in the form of natural language, touch coordinates or stylus coordinates acquired via a touch panel or digitizer included in the display, images and / or videos, but is not limited thereto. In addition, context information may also be transmitted when transmitting the user input. The context information may be user input. It can include various additional information at the time point. For example, the additional information can include information on the application currently being used by the user or the user's location information. In addition, the user input can be a mixed form of the above-described natural language, images, sounds, and context information. In addition, the user input can also be a non-natural language form, such as selecting a menu. The user query / response interface (210) can output the results of the generative artificial intelligence system (200) and / or the results of analyzing the input to the user. The output can be in the form of natural language or a specific content, and can also be provided in the form of an action requested by the user. The user query / response interface (210) can output the results of the generative artificial intelligence system (200) to the user. The output can be in the form of natural language or a specific content, and can also be provided in the form of an action requested by the user.

[0059] According to one embodiment, the AI ​​framework (240) can receive user input and coordinate and control each component necessary to perform the user's intention based on the user's query.

[0060] According to one embodiment, user input received from the user query / response interface (210) may be transmitted to a prompt design component (241). The prompt design component (241) may be used to generate prompts suitable for inputting the user input into a large language model (LLM) or a large multimodal model (LMM). The prompt design component (241) may be an AI component that uses a machine learning algorithm or a neural network to develop better prompts over time. The prompt design component (241) may access a knowledge component including user preference data, a prompt library, and prompt examples based on the user input to generate prompts, and transmit the generated prompts to the LLM or LMM.

[0061] According to one embodiment, the API / Plug-in management component (242) may communicate with external information when there is a request for additional information when passing user input as input to the generative model. The API / Plug-in management component (242) may establish a channel for communicating with the outside of the AI ​​Interface through the API, and may enable access to various data sources (e.g., knowledge repositories (220)) through the established channel. In addition, the API / Plug-in management component (242) may request the application / service component (230) through the API to perform an action that ultimately performs the user input, rather than an intermediate result, when the action needs to be performed in the application or service. Information obtained from the outside may be used to generate a prompt in the prompt design component (241) together with the user input, or may be passed as an input to the generative model.

[0062] According to one embodiment, the output modification component (also referred to as a refiner component) (243) can fine-tune the output from the generative model. For example, the output modification component (243) can verify that the content generated through the LLM and / or LMM is not irrelevant, does not contain biased content, or does not contain harmful content. In addition, the output modification component (243) can determine to what extent the content matches the result desired by the user and, if necessary, can perform additional processing. The output modification component (243) can additionally configure and provide the user with hints to avoid undesired output.

[0063] According to one embodiment, a generative AI model (260) may generally refer to an artificial intelligence neural network that creates new types of data based on user input information. The generative AI model (260) may include a model that generates images and / or a model that generates languages. Representative models that generate images include a generative adversarial network (GAN) and a variational auto encoder (VAE), and examples include a diffusion-based generative model that uses a VAE and a Transformer structure. A model that generates languages ​​is a model that is trained to statistically output the most appropriate output based on input values, and representative examples include models such as CHAT-GPT 3 and CHAT-GPT 4. In addition, there are also large multimodal models (LMMs) that can recognize various types of data input, such as text, images, and voice, and generate new data corresponding to them.

[0064] FIG. 2 is a diagram illustrating an operation of monitoring a user and acquiring sensing data according to one embodiment of the present disclosure.

[0065] The components and operations of the components described with reference to FIG. 2 may be partially or entirely the same as the components and operations of the components described with reference to FIG. 1a and FIG. 1b.

[0066] Referring to FIG. 2, according to one embodiment, at least one electronic device can continuously monitor a user (U) to obtain sensing data. The at least one electronic device may include, but is not limited to, a mobile electronic device (300) (e.g., a mobile robot or a smart home robot) and / or a wearable electronic device (400) (e.g., a smart watch, a smart ring, or a head mounted display). For example, the electronic device (101) of FIG. 1A or at least one electronic device (e.g., a smart speaker, an intelligent camera) placed at a location where the user is located may be used to monitor the user (U).

[0067] According to one embodiment, a mobile electronic device (300) can monitor a user (U) using at least one sensor to obtain first sensing data. The mobile electronic device (300) can search for a user (U) requiring monitoring, move to a location adjacent to the searched user (U), and monitor the user (U).

[0068] According to one embodiment, the first sensing data may include at least one first modality data associated with a user (U). The at least one first modality data may include, but is not limited to, modality data including image data of the user (U), modality data including voice or sound data of the user, modality data including location data of the user, and / or modality data including movement or motion data of the user.

[0069] According to one embodiment, a wearable electronic device (400) can monitor a user (U) using at least one sensor to obtain second sensing data. The wearable electronic device (400) can be worn on a part of the user's (U's) body (e.g., wrist, finger, head) to monitor the user (U).

[0070] According to one embodiment, the second sensing data may include at least one second modality data associated with the user (U). The at least one second modality data may include, but is not limited to, modality data including image data of the user (U), modality data including voice or sound data of the user, modality data including location data of the user, modality data including movement or motion data of the user, and / or modality data including biometric data of the user (e.g., pulse, blood pressure, electrocardiogram, brain wave).

[0071] According to one embodiment, an electronic device may use multimodal data including first sensed data and / or second sensed data to support associative learning for a user. The electronic device may assist in the user's memory recovery and enhancement by supporting associative learning.

[0072] FIG. 3A is a diagram illustrating a mobile electronic device according to one embodiment of the present disclosure.

[0073] FIG. 3b is a block diagram of the mobile electronic device of FIG. 3a, according to one embodiment of the present disclosure.

[0074] The components and operations of the components described with reference to FIGS. 3a and 3b may be partially or entirely identical to the components and operations of the components described with reference to FIGS. 1 and 2.

[0075] Referring to FIGS. 3A and 3B , according to one embodiment, a mobile electronic device (300) may include a main body (301), a driving module (311), a driving module (312), a communication module (320), a display (331), a projector (332), a sensor module (340), at least one camera (350), at least one microphone (361), at least one speaker (362), a power module (370), at least one memory (380), and / or at least one processor (390). According to one embodiment, the mobile electronic device (300) may include additional components (e.g., an input / output interface) other than the illustrated components, or may omit at least one of the illustrated components.

[0076] According to one embodiment, the body (301) may form the exterior of the mobile electronic device (300). For example, the body (301) may form a cylindrical or spherical exterior, but is not limited thereto and may also form an exterior of another shape, such as a hexahedron. The body (301) may be configured to be rotatable. For example, the body (301) may be rotated by a motor and may be rotated based on a preset angle or direction. The motors may be provided on each of both sides of the body (301). In the present disclosure, the body (301) may also be referred to as a housing.

[0077] According to one embodiment, the drive module (311) may include at least one motor (e.g., a dual motor) used to move the mobile electronic device (300) or the main body (301). According to one embodiment, the drive module (312) may include at least one wheel used to move the mobile electronic device (300) or the main body (301). For example, the drive module (312) may include a first wheel and a second wheel installed on both lower sides of the main body (301), and the drive module (311) may include two wheel motors for rotating the first wheel and the second wheel, respectively. The first wheel and the second wheel may have corresponding configurations and may be arranged symmetrically with respect to the main body (301).

[0078] According to one embodiment, the communication module (320) includes at least one communication circuit and can support the establishment of a wired communication channel or a wireless communication channel between the mobile electronic device (300) and an external electronic device (e.g., the electronic device (101), the server (108) of FIG. 1A, the wearable device (400) of FIG. 2), and the performance of communication through the established communication channel.

[0079] According to one embodiment, the communication module (320) may include a wired communication module and / or a wireless communication module. For example, the wired communication module may include a local area network (LAN) communication module or a power line communication module, and the wireless communication module may include a short-range communication module and / or a mobile communication module.

[0080] According to one embodiment, the short-range communication module may include at least one of a near field communication (NFC) communication module, a radio-frequency identification (RFID) communication module, a Zigbee communication module, a wireless fidelity (Wi-Fi) communication module, a Wi-Fi Direct communication module, a Bluetooth communication module, a Bluetooth low energy (BLE) communication module, a wireless local area network (WLAN) communication module, an Ant+ communication module, an ultra wideband (UWB) communication module, an infrared data association (IrDA) communication module, or a microwave (uWave) communication module.

[0081] According to one embodiment, the mobile communication module may include at least one communication module capable of communicating with an external electronic device via a long-distance communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or a wide area network (WAN)).

[0082] According to one embodiment, the communication module (320) may include or be connected to an antenna module used to transmit or receive signals or power to or from an external electronic device. The antenna module may include one or more antennas (e.g., an array antenna).

[0083] In one embodiment, the display (331) can visually provide information to an external device (e.g., a user) of the mobile electronic device (300). The display (331) can include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. In one embodiment, the display (331) can include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0084] According to one embodiment, the projector (332) may be configured to be included inside the main body (301) and project an image through a hole formed in a portion (e.g., the front) of the main body (301). The projector (332) may project an image onto various projection surfaces (e.g., a wall, a screen, a floor) based on a rotation angle or rotation direction of the main body (301). The image may include, for example, at least one of a still image, a moving image, a real-time streaming image, a broadcast image, a simulation image, an interactive image, or an animation image.

[0085] According to one embodiment, the sensor module (340) may include a plurality of sensors configured to detect information about the surrounding environment of the mobile electronic device (300). For example, the sensor module (340) may include a depth sensor (341), an obstacle sensor (342), and / or an inertial sensor (343). The depth sensor (341) may detect depth or distance. The depth sensor (341) may include, for example, a time-of-flight (ToF) sensor and / or a lidar sensor. The obstacle sensor (342) may recognize an obstacle located in the surroundings (e.g., in front) of the mobile electronic device (300). The obstacle sensor (342) may include, for example, an IR sensor and / or an ultrasonic sound sensor. The inertial sensor (343) may detect a posture and / or movement of the mobile electronic device (300). The inertial sensor (343) may include, for example, an accelerometer sensor and / or a gyro sensor. The sensor module (340) may further include at least one additional sensor in addition to the sensors illustrated. For example, the sensor module (340) may further include an illuminance sensor that measures ambient brightness.

[0086] According to one embodiment, the camera (350) is located inside the main body (301) and can convert light input through a hole formed in a part (e.g., the front) of the main body (301) into an electrical signal. The mobile electronic device (300) can capture still images or moving images through the camera (350). The mobile electronic device (300) can include at least one camera. The at least one camera can include, but is not limited to, an RGB (red green blue) camera, a wide camera, and / or an ultra-wide camera.

[0087] According to one embodiment, the microphone (361) is located inside the main body (301), and can receive sound through a hole formed in a part of the main body (301) and convert the received sound into an electric signal. According to one embodiment, the mobile electronic device (300) can obtain a user's voice input or ambient sound information through the microphone (361). The mobile electronic device (300) can include a plurality of microphones (361) that are spaced apart from each other. When a plurality of microphones (361) are included in the mobile electronic device (300), the mobile electronic device (300) can better obtain the user's voice even in a noisy environment through beamforming technology.

[0088] According to one embodiment, the speaker (362) is located inside the main body (301) and can convert an electrical signal into sound. The speaker (362) can output sound through a hole formed in a part of the main body (301). According to one embodiment, the mobile electronic device (300) can provide audible notifications or information to the user through the speaker (362). The mobile electronic device (300) can include a plurality of speakers (362) spaced apart from each other.

[0089] According to one embodiment, the power module (370) may include a device or circuit that supplies power to the mobile electronic device (300). The power module (370) may include an interface (372) (e.g., a wireless charging interface) that can receive power from an external power supply device, and a battery (371) that stores power received from the external power supply device.

[0090] According to one embodiment, the memory (380) may store various data that may be used to control the operation of each component of the mobile electronic device (300). The memory (380) may include, for example, at least one storage medium that stores a plurality of application programs used in the mobile electronic device (300), data for controlling the operation of the mobile electronic device (300), and commands. The commands stored in the memory (380), when executed by at least one processor (390), may cause the mobile electronic device (300) to perform at least one operation (e.g., at least one of the operations to be described later in FIGS. 5 to 14). At least some of the application programs stored in the memory (380) may be downloaded from an external source (e.g., a server) via wireless communication. At least some of the application programs stored in the memory (380) may be stored in the memory (380) from the time of shipment for the basic functions of the mobile electronic device (300).

[0091] According to one embodiment, the memory (380) can store at least one program for processing and controlling the processor (390), and can store input and / or output data. The memory (380) can also store at least one artificial intelligence (AI) model. The memory (380) can include at least one of a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD (secure digital) or XD (extreme digital) memory), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, and an optical disk. According to one example, a web storage or a cloud server that performs a storage function on the Internet can be operated by the mobile electronic device (300).

[0092] According to one embodiment, at least one processor (390) can control the overall operation of the mobile electronic device (300) and perform operations of the mobile electronic device (300) to be presented below. The processor (390) can execute calculations or data processing related to control and / or communication of at least one other component of the mobile electronic device (300). For example, the processor (390) is electrically connected to the drive module (311), the driving module (312), the communication module (320), the display (331), the projector (332), the sensor module (340), the camera (350), the microphone (361), the speaker (362), and / or the memory (380), and can execute instructions stored in the memory (380). The processor (390) can include at least one processing circuit that executes instructions stored in the memory (380).

[0093] In one embodiment, at least one processor (390) may include various processing circuits and / or multiple processors. One or more of the at least one processor (390) may be individually and / or collectively configured to perform various functions described herein. In this disclosure, when "a processor," "at least one processor," and "one or more processors" are described as being configured to perform various functions, these terms include, but are not limited to, situations where one processor performs some of the recited functions and other processor(s) perform other parts of the recited functions, and also situations where a single processor can perform all of the recited functions. Additionally, the at least one processor (390) may include a combination of processors that perform the various recited / disclosed functions, for example, in a distributed manner. The at least one processor (390) may execute program instructions to achieve or perform the various functions.

[0094] According to one embodiment, at least one processor (390) may include at least one of a central processing unit (CPU), an NPU, a graphics processing unit (GPU), a micro processing unit (MPU), a micro controller unit (MCU), an application processor (AP), a communication processor (CP), a system on chip (SoC), or an integrated circuit (IC), a sensor hub, a supplementary processor, an application specific integrated circuit (ASIC), or a field programmable gate array (FPGA), and may have multiple cores.

[0095] FIG. 4A is a diagram illustrating a front side of a wearable electronic device according to one embodiment of the present disclosure.

[0096] FIG. 4B is a drawing showing the back of the wearable device of FIG. 4A according to one embodiment of the present disclosure.

[0097] FIG. 4C is a block diagram of the wearable device of FIG. 4A, according to one embodiment of the present disclosure.

[0098] The components and operations of the components described with reference to FIGS. 4a to 4c may be partially or entirely identical to the components and operations of the components described with reference to FIGS. 1 to 3.

[0099] Referring to FIGS. 4A, 4B, and 4C, a wearable electronic device (400) (e.g., a watch-type electronic device) according to one embodiment may include a housing (401), a communication module (410), a display (420), a sensor module (430), at least one camera (440), at least one microphone (450), at least one speaker (460), a power module (470), at least one memory (480), and / or at least one processor (490). According to one embodiment, the wearable electronic device (400) may include additional components (e.g., an input / output interface) other than the illustrated components, or may omit at least one of the illustrated components.

[0100] According to one embodiment, the housing (401) may include a first side (or front side) (401A), a second side (or back side) (401B), and a side surface (401C) enclosing a space between the first side (401A) and the second side (401B).

[0101] According to one embodiment, the communication module (410) includes at least one communication circuit and can support the establishment of a wired communication channel or a wireless communication channel between the wearable electronic device (400) and an external electronic device (e.g., the electronic device (101), the server (108) of FIG. 1A, the mobile electronic device (300) of FIG. 2), and the performance of communication through the established communication channel.

[0102] According to one embodiment, the communication module (410) may include a wired communication module and / or a wireless communication module. For example, the wired communication module may include a LAN communication module or a power line communication module, and the wireless communication module may include a short-range communication module and / or a mobile communication module.

[0103] According to one embodiment, the short-range communication module may include at least one of an NFC communication module, an RFID communication module, a Zigbee communication module, a Wi-Fi communication module, a Wi-Fi Direct communication module, a Bluetooth communication module, a BLE communication module, a WLAN communication module, an Ant+ communication module, a UWB communication module, an IrDA communication module, or a microwave communication module.

[0104] According to one embodiment, the mobile communication module may include at least one communication module capable of communicating with an external electronic device via a long-distance communication network (e.g., a LAN or WAN), such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network.

[0105] According to one embodiment, the communication module (410) may include or be connected to an antenna module used to transmit or receive signals or power to or from an external electronic device. The antenna module may include one or more antennas (e.g., an array antenna).

[0106] According to one embodiment, the display (420) can visually provide information to an external party (e.g., a user) of the wearable electronic device (300). The display (420) may have a shape corresponding to the shape of the front surface (401A), and may have various shapes such as a circle, an oval, or a polygon. The display (420) may be coupled to or disposed adjacent to a touch detection circuit, a pressure sensor capable of measuring the intensity (pressure) of a touch, and / or a fingerprint sensor.

[0107] According to one embodiment, the sensor module (430) may include a plurality of sensors configured to detect information about the surrounding environment of the wearable electronic device (400). The sensor module (430) may generate an electrical signal or a data value corresponding to an internal operating state of the wearable electronic device (400) or an external environmental state. The sensor module (431) may include, for example, an inertial sensor (431) (e.g., an accelerometer, a gyro sensor, a barometer) disposed inside the housing (401), and at least one biometric sensor disposed on one surface (e.g., a side surface (410C) or a rear surface (401B)) of the housing (401). The at least one biometric sensor may include, but is not limited to, a first electrode sensor (432a), a second electrode sensor (432b), a third electrode sensor (432c), a photoplethysmogram (PPG) sensor (433), and / or a temperature sensor (434). For example, electrode sensors (432a, 432b, 432c) can be used to detect electrical signals to monitor first biosignals (e.g., electrocardiogram, electroencephalogram, electromyogram). For example, a PPG sensor can be used to detect changes in blood flow using light to monitor second biosignals (e.g., heart rate, oxygen saturation). For example, a temperature sensor (434) can be used to detect electrical signals to measure third biosignals (e.g., skin temperature).

[0108] According to one embodiment, the camera (440) is positioned inside the housing (401) and can convert light input through a hole formed in a portion (e.g., the front) of the housing (401) into an electrical signal. The wearable electronic device (400) can capture still images or videos through the camera (440). The wearable electronic device (400) can include at least one camera. The at least one camera can include, but is not limited to, an RGB camera, a depth camera, a wide camera, and / or an ultra-wide camera.

[0109] According to one embodiment, the microphone (450) is located inside the housing (401), and can receive sound through a hole formed in a part of the housing (401) and convert the received sound into an electrical signal. According to one embodiment, the wearable electronic device (400) can obtain a user's voice input or ambient sound information through the microphone (450). The wearable electronic device (400) can include a plurality of microphones (e.g., a first microphone (451) and a second microphone (452)) that are spaced apart from each other. When a plurality of microphones are included in the wearable electronic device (400), the wearable electronic device (400) can better obtain the user's voice even in a noisy environment through beamforming technology.

[0110] In one embodiment, the speaker (460) is located inside the housing (401) and can convert an electrical signal into sound. The speaker (460) can output sound through a hole formed in a portion of the housing (401). In one embodiment, the wearable electronic device (400) can audibly provide a notification or information to the user through the speaker (460).

[0111] According to one embodiment, the power module (470) may include a device or circuit that supplies power to the wearable electronic device (400). The power module (470) may include an interface (e.g., a wireless charging interface) that can receive power from an external power supply device, and a battery that stores power received from the external power supply device.

[0112] According to one embodiment, the memory (480) may store various data that may be used to control the operation of each component of the wearable electronic device (400). The memory (480) may include, for example, at least one storage medium that stores a plurality of application programs used in the wearable electronic device (400), data for controlling the operation of the wearable electronic device (400), and commands. The commands stored in the memory (480), when executed by at least one processor (490), may cause the wearable electronic device (400) to perform at least one operation (e.g., at least one of the operations of FIGS. 5 to 14). At least some of the application programs stored in the memory (480) may be downloaded from an external source (e.g., a server) via wireless communication. At least some of the application programs stored in the memory (480) may be stored in the memory (480) from the time of shipment for the basic functions of the wearable electronic device (400).

[0113] According to one embodiment, the memory (480) can store at least one program for processing and controlling the processor (490) and can store input and / or output data. The memory (480) can also store at least one AI model. The memory (480) can include at least one of a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory), a RAM, an SRAM, a ROM, an EEPROM, a PROM, a magnetic memory, a magnetic disk, and an optical disk. According to one example, a web storage or cloud server that performs a storage function on the Internet may be operated by the wearable electronic device (400).

[0114] According to one embodiment, at least one processor (490) may control the overall operation of the wearable electronic device (400) and may perform operations of the wearable electronic device (400) to be presented below. The processor (490) may execute calculations or data processing related to control and / or communication of at least one other component of the wearable electronic device (400). For example, the processor (490) may be electrically connected to a communication module (410), a display (420), a sensor module (430), a camera (440), a microphone (450), a speaker (460), a power module (470), and / or a memory (480), and may execute instructions stored in the memory (480). The processor (490) may include at least one processing circuit that executes instructions stored in the memory (480).

[0115] According to one embodiment, at least one processor (490) may include various processing circuits and / or multiple processors. One or more of the at least one processor (490) may be individually and / or collectively configured to perform various functions described herein. In this disclosure, when "a processor," "at least one processor," and "one or more processors" are described as being configured to perform various functions, these terms include, but are not limited to, situations where one processor performs some of the recited functions and other processor(s) perform other parts of the recited functions, and also situations where a single processor can perform all of the recited functions. Additionally, the at least one processor (490) may include a combination of processors that perform the various recited / disclosed functions, for example, in a distributed manner. The at least one processor (490) may execute program instructions to achieve or perform the various functions.

[0116] According to one embodiment, at least one processor (490) may include at least one of a CPU, an NPU, a GPU, an MPU, an MCU, an AP, a CP, an SoC, an IC sensor hub, a coprocessor, an ASIC, or an FPGA, and may have multiple cores.

[0117] FIG. 5 is a flowchart illustrating a method by which an electronic device supports associative learning according to one embodiment of the present disclosure.

[0118] FIG. 6 is a diagram illustrating a configuration of an electronic device for supporting associative learning according to one embodiment of the present disclosure.

[0119] FIG. 7 is a diagram for explaining the configuration of a multimodal model according to one embodiment of the present disclosure.

[0120] The components and operations of the components described with reference to FIGS. 5 to 7 may be partially or entirely identical to the components and operations of the components described with reference to FIGS. 1 to 4.

[0121] For convenience of explanation, in the embodiments of FIGS. 5 to 7, the electronic device is described as an example of a mobile electronic device (300) of FIGS. 2, 3a, and 3b. However, the embodiments are not limited thereto, and the description of the embodiments of FIGS. 5 to 7 may also be applied to other types of electronic devices (e.g., the electronic device (101) of FIG. 1a, the server (108) of FIG. 1a, or the wearable electronic device (400) of FIGS. 2, 4a to 4c). In the following embodiments, the operation of the electronic device may be understood as being performed by at least one processor included in the electronic device (e.g., the processor (390) of FIGS. 3a and 3b).

[0122] In the following examples, the operations may be performed sequentially, but are not necessarily sequential. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0123] Referring to FIGS. 6 and 7, according to one embodiment, the electronic device may include an auto speech recognition (ASR) module (610), a multi-modal sensing module (620), a multi-modal module (630), and / or a language module (640). According to one embodiment, the operation of each module included in the electronic device may be implemented by at least one memory and at least one processor, respectively.

[0124] According to one embodiment, the ASR module (610) may perform a function of converting a voice signal (e.g., a first user voice input (611) and / or a second user voice input (612a, 612b; hereinafter, collectively referred to as 612)) into text and transmitting the converted text to the multimodal module (630) and / or the language module (640). The ASR module (610) may include at least one AI model (e.g., an ASR model) for converting a voice signal into text. The ASR model may convert a voice signal into text. According to one embodiment, when the user input is not a voice input, the user input (e.g., a text input) may bypass the ASR module (610) and be transmitted to the multimodal module (630) and / or the language module (640).

[0125] According to one embodiment, the multimodal sensing module (620) may collect sensing data (e.g., first sensing data (621) and / or second sensing data (622)) associated with a user (e.g., user (U) of FIG. 2) and transmit the sensing data or at least one modality data included in the sensing data (e.g., modality data (701 to 705)) to the multimodal module (630). According to one embodiment, the multimodal sensing module (620) may extract features for each modality data and transmit information about the extracted features to the multimodal module (630).

[0126] According to one embodiment, the multimodal module (630) may be configured to generate scene data (631), information related to a user's behavioral pattern (632), and / or correct answer information (633). The scene data (631), information related to a user's behavioral pattern (632), and / or correct answer information (633) are described below with reference to FIG. 5.

[0127] According to one embodiment, the multimodal module (630) may include at least one multimodal AI model (700) for generating scene data (631), information related to a user's behavioral pattern (632), and / or correct answer information (633). The multimodal AI model (700) may be an AI model capable of simultaneously processing and understanding various types of data (multimodal data). The multimodal module (630) may include a first multimodal AI model trained to generate scene data (631) based on first input data (e.g., input data including first multimodal data), a second multimodal AI model trained to generate correct answer information (633) based on second input data (e.g., input data including second multimodal data), and / or a third multimodal AI model trained to generate information related to a user's behavioral pattern (632) based on third input data (e.g., input data including third multimodal data). According to one embodiment, the multimodal AI model (700) may be an AI model included in a second generative AI model (e.g., a generative AI model (260) corresponding to the LMM of FIG. 1b). The multimodal module (630) may include a prompt module (e.g., a prompt design component (241) of FIG. 1b) for generating prompt data to be input to the multimodal AI model based on input data input to the multimodal module (630). In the present disclosure, the multimodal AI model (700) may also be referred to as a multimodal model.

[0128] According to one embodiment, as illustrated in FIG. 7, a multimodal AI model (700) may include at least one feature extraction module (711 to 715) for extracting features for at least one modality data (701 to 705) provided through an input layer, an encoder (720), at least one dense layer (730), and / or an output layer (740).

[0129] According to one embodiment, each feature extraction module can extract features for the corresponding modality data. For example, at least one feature extraction module can include a first feature extraction module (711) that extracts features for first modality data (701) including motion or movement data, a second feature extraction module (712) that extracts features for second modality data (702) including biosignal data, a third feature extraction module (713) that extracts features for third modality data (703) including position data, a fourth feature extraction module (714) that extracts features for fourth modality data (704) including image data, and / or a fifth feature extraction module (715) that extracts features for voice or sound data (705).

[0130] In one embodiment, the encoder (720) can integrate features extracted from each modality data to generate a multimodal representation representing the integrated features. The encoder (720) can generate the multimodal representation using, for example, a transformer-based model. The encoder (720) may be referred to as a multimodal encoder.

[0131] In one embodiment, the dense layer (730) may be a layer used to generate a final output based on the multimodal representation generated by the encoder (720). The multimodal model may learn complex patterns through at least one dense layer (730). The dense layer (730) may also be referred to as a fully connected layer.

[0132] In one embodiment, the output layer (740) may output different information depending on the characteristics of the task. For example, in the case of a generation task, the output layer (740) may include a decoder for generating text or an image, and may output the generated text or an image.

[0133] According to one embodiment, the language module (640) may be responsible for generating hint information (641a, 641b; hereinafter collectively referred to as 641) and / or correct answer evaluation information (642). The hint information (641) and / or correct answer evaluation information (642) are described below with reference to FIG. 5.

[0134] According to one embodiment, the language module (640) may include at least one language AI model for generating hint information (641) and / or correct answer evaluation information (642). For example, the language module (640) may include a first language AI model trained to generate hint information (641) based on first input data (e.g., first input data including correct answer information (633)) and / or a second language AI model trained to generate correct answer evaluation information (642) based on second input data (e.g., user input including correct answer information (633) and a response to the hint information (641). According to one embodiment, the language model may be an AI model included in the first generative AI model (e.g., the generative AI model (260) corresponding to the LLM of FIG. 1B). The language module (640) may include a prompt module (e.g., the prompt design component (241) of FIG. 1B) for generating prompt data to be input to the language AI model based on input data input to the language module (640). In the present disclosure, the language AI model may also be referred to as a language model.

[0135] In the present disclosure, inputting at least one piece of information / data into an AI model (e.g., a first generative AI model (e.g., LLM), a second generative AI model (e.g., LMM), a multimodal AI model, a language AI model) can be understood as inputting prompt data generated based on the at least one piece of information / data into the AI ​​model.

[0136] Referring to FIGS. 5 and 6, according to one embodiment, in operation 510, the electronic device may obtain a first user input of a user (e.g., the user (U) of FIG. 2). The first user input may include, but is not limited to, a first user voice input (611). For example, the first user input may include a text input, a gesture input, or a video input. In the embodiment of FIGS. 5 and 6, the first user input refers to a first input of a user (e.g., the user (U) of FIG. 2), and the second user input refers to a second input of a user who is the same user as the user of the first user input (e.g., the user (U) of FIG. 2). The first user voice input (611) refers to a first voice input of a user (e.g., the user (U) of FIG. 2), and the second user input (612) refers to a second voice input of a user who is the same user as the user of the first user voice input (e.g., the user (U) of FIG. 2).

[0137] According to one embodiment, when the first user input includes a first user voice input (611), the electronic device may convert a voice signal corresponding to the first user voice input (611) into text using the ASR module (610). The ASR module (610) may convert the voice signal corresponding to the first user voice input (611) into text and transmit the converted text to the multimodal module (630).

[0138] According to one embodiment, in operation 520, the electronic device may generate correct information (633) for information corresponding to the first user input. The information corresponding to the first user input may include, for example, information that the user wishes to remember (e.g., information (1210) of FIG. 12A or information (1310) of FIG. 13A).

[0139] According to one embodiment, the electronic device may generate scene data (631) based on sensing data, and generate correct information (633) for information corresponding to the first user input based on the generated scene data (631). According to one embodiment, the scene data (631) may be associated with a user's action.

[0140] According to one embodiment, the sensing data may include first sensing data (621) acquired from an electronic device and / or second sensing data (622) acquired from an external electronic device (e.g., a wearable device (400) of FIG. 2).

[0141] According to one embodiment, the first sensing data (621) may include at least one first modality data associated with the user. The at least one first modality data may include, but is not limited to, modality data including image data of the user, modality data including voice or sound data of the user, modality data including location data of the user, and / or modality data including movement or motion data of the user.

[0142] According to one embodiment, the second sensing data (622) may include at least one second modality data associated with the user. The at least one second modality data may include, but is not limited to, modality data including the user's voice or sound data, modality data including the user's location data, modality data including the user's movement or motion data, and / or modality data including the user's biometric data (e.g., pulse, blood pressure, electrocardiogram, brain waves).

[0143] According to one embodiment, the electronic device may acquire or collect sensing data including first sensing data (621) and / or second sensing data (622) using a multimodal sensing module (620). The multimodal sensing module (620) may transmit the collected sensing data or at least one modality data included in the collected sensing data to the multimodal module (630).

[0144] According to one embodiment, scene data (631) may include information (e.g., summary information) about at least one scene associated with a session. An example of scene data (631) is described below with reference to FIG. 9A.

[0145] According to one embodiment, an electronic device may generate scene data (631) based on sensing data that may be grouped into a single session (or group). A session may be based on, for example, a specific activity of a user, a specific location, a specific event, and / or a specific time. For example, sensing data associated with a specific activity of a user may be grouped into a single session. For example, sensing data for the time a user stayed at a specific location may be grouped into a single session. For example, sensing data associated with a specific event may be grouped into a single session. In the present disclosure, sensing data grouped into a single session may be referred to as session sensing data. Each session sensing data may include at least a portion of the sensing data collected through the multimodal sensing module (620). Operations for obtaining session sensing data (e.g., segmentation and sessionization) are described below with reference to FIGS. 8 to 11 .

[0146] According to one embodiment, the electronic device may obtain scene data (631) using a multimodal module (630). The multimodal module (630) may obtain sensing data (e.g., session sensor data), extract feature information for each modality data included in the sensing data, and generate scene data (631) based on the feature information. The multimodal module (630) may generate scene data (631) using a first multimodal AI model (e.g., a first AI model included in a second generative AI model (e.g., LMM)) based on the feature information. As described above, the first multimodal AI model may be an AI model trained to generate scene data (631).

[0147] According to one embodiment, the electronic device may generate correct answer information (633) using a multimodal module (630). For example, the multimodal module (630) may input scene data (631) and information corresponding to a first user input into a second multimodal AI model (e.g., a second AI model included in a second generative AI model (e.g., LMM)), and obtain correct answer information (633) for the information corresponding to the first user input from the second multimodal AI model. As described above, the second multimodal AI model may be an AI model trained to generate correct answer information (633).

[0148] According to one embodiment, the electronic device may transmit at least a portion of the generated scene data to the server (650). Since the amount of generated scene data may continuously increase over time, if the memory of the electronic device is insufficient, the electronic device may transmit at least a portion of the generated scene data to the server (650) so that the scene data may be managed by the server (650). When transmitting the scene data to the server (650), the electronic device may encrypt and compress the transmitted scene data to protect personal information and transmit it. When necessary, the electronic device may request the scene data managed by the server (650) to the server, and receive and use the requested scene data from the server (650).

[0149] According to one embodiment, in operation 530, the electronic device may generate first hint information (641) associated with the correct answer information (633) using a first generative AI model (e.g., a language AI model) based on correct answer information (633) and / or information related to a user's behavioral pattern (632), and provide the first hint information (641). According to one embodiment, the first hint information (641) may include first information that helps to associate the correct answer information (633). The first hint information (641) may be used for associative learning.

[0150] According to one embodiment, the electronic device may determine whether to use information (632) related to the user's behavioral pattern as source data for generating hint information based on information corresponding to the first user response (e.g., information that the user wishes to remember). For example, if the information that the user wishes to remember corresponds to, for example, information (1210) of FIG. 12A, the electronic device may not use information (632) related to the user's behavioral pattern as source data for generating hint information. For example, if the information that the user wishes to remember corresponds to, for example, information (1310) of FIG. 13A, the electronic device may use information (632) related to the user's behavioral pattern as source data for generating hint information. When information (632) related to the user's behavioral pattern is used as source data for generating hint information, information related to the user's behavior (e.g., the user's lifestyle habits, behavioral patterns, and daily activities) corresponding to the information (632) related to the user's behavioral pattern is taken into consideration when generating hint information, so that more accurate and useful hint information for associating correct answer information can be generated. In this disclosure, information (632) related to a user's behavioral pattern is used as source data for generating hint information, but this is not limited thereto. For example, various types of contextual information related to the user may be used as source data for generating hint information. For example, information related to the user's personal tastes and preferences may be used as source data for generating hint information. In this disclosure, information (632) related to a user's behavioral pattern may also be referred to as "behavior pattern-related information" or "user behavior pattern-related information."

[0151] According to one embodiment, the electronic device may generate information (632) related to a user's behavior pattern based on scene data (631). For example, the electronic device may generate information (632) related to a user's behavior pattern based on scene data (631) using a multimodal module (630). According to one embodiment, the multimodal module (630) may generate information (632) related to a user's behavior pattern based on at least one scene data (631) (or a scene data set including at least one scene data (631)) collected during a specified period (e.g., 1 day). For example, the multimodal module (630) may input at least one scene data (631) or a scene data set collected during a specified period into a third multimodal AI model (e.g., a third AI model included in a second generative AI model (e.g., LMM)) and obtain information (632) related to the user's behavior pattern from the third multimodal AI model. As described above, the third multimodal AI model may be an AI model trained to generate information (632) related to a user's behavioral pattern. An example of information (632) related to a user's behavioral pattern is described below with reference to FIG. 9B.

[0152] According to one embodiment, information (632) related to a user's behavioral pattern may include at least one behavioral element. The at least one behavioral element may include, for example, at least one first behavioral element related to a temporal element (e.g., time, frequency), at least one second behavioral element related to a spatial element (e.g., place, location), and / or at least one third behavioral element related to an environmental (or situational) element (e.g., surroundings, interactions, surrounding people). In the present disclosure, a behavioral element may also be referred to as behavioral information.

[0153] According to one embodiment, the electronic device may use the language module (640) to generate first hint information (641a) associated with the correct answer information (633) based on the correct answer information (633) and / or information related to the user's behavior pattern (632). For example, the language module (640) may input the correct answer information (633) and the information related to the user's behavior pattern (632) into a first language AI model (e.g., a first AI model included in a first generative AI model (e.g., LLM)) and obtain first hint information (641a) associated with the correct answer information (633) from the first language AI model. As described above, the first language AI model may be an AI model trained to generate hint information.

[0154] According to one embodiment, the electronic device can visually provide the first hint information (641a) via a display (e.g., the display (331) of FIGS. 3A and 3B) or a projector (e.g., the projector (332) of FIGS. 3A and 3B). For example, the electronic device can display the first hint information (641a) via a display. For example, the electronic device can project the first hint information (641a) onto a projection surface (e.g., a floor, a wall, a screen) via a projector.

[0155] In one embodiment, the electronic device may audibly provide the first hint information (641a) via a speaker (e.g., speaker (362) of FIGS. 3A and 3B ). For example, the electronic device may output a sound corresponding to the first hint information (641a) via the speaker.

[0156] According to one embodiment, in operation 540, the electronic device may obtain a second user input (e.g., a second user input including a first response to the first hint information (641a)) of a user (e.g., a user (U) of FIG. 2) associated with the first hint information (641a). The second user input may include, but is not limited to, a second user voice input (612a, 612b; 612). For example, the second user input may include a text input, a gesture input, or a video input.

[0157] According to one embodiment, when the second user input includes a second user voice input (612), the electronic device may convert a voice signal corresponding to the second user voice input (612) into text using the ASR module (610). The ASR module (610) may convert the voice signal corresponding to the second user voice input (612) into text and transmit the converted text to the language module (640).

[0158] According to one embodiment, the first response may include information that the user has remembered and / or information requesting confirmation of the information that the user has remembered through the first hint information (641a).

[0159] According to one embodiment, the electronic device may identify a user input obtained within a specified time (e.g., 5 seconds) after the first hint information (641a) is provided as a second user input including a first response to the first hint information (641a). According to one embodiment, the electronic device may identify a user input including information requesting confirmation of information remembered by the user through the first hint information (641a) as a second user input including a first response to the first hint information (641a).

[0160] According to one embodiment, in operation 550, the electronic device may evaluate a first response based on correct answer information (633). For example, the electronic device may evaluate the first response based on first similarity information indicating a similarity score or value between the correct answer information (633) and the first response.

[0161] According to one embodiment, the electronic device may obtain (or generate) correct answer evaluation information (642) for the first response based on correct answer information (633) using the language module (640), and may evaluate the first response based on the correct answer evaluation information (642). For example, the language module (640) may input correct answer information (633), information related to a user's behavioral pattern (632), a first user input (612a) including the first response, and / or first hint information (641a) into a second language AI model (e.g., a second AI model included in a first generative AI model (e.g., LLM)), and obtain correct answer evaluation information (642) for the first response from the second language AI model. As described above, the second language AI model may be an AI model trained to generate correct answer evaluation information (e.g., a similarity score).

[0162] According to one embodiment, the correct answer evaluation information (642) for the first response may include first similarity information indicating a similarity score between the correct answer information (633) and the first response. When the correct answer evaluation information (642) includes the first similarity information, the electronic device may evaluate whether the first response is correct based on the first similarity information included in the correct answer evaluation information (642). For example, if the similarity score of the first similarity information is equal to or greater than a reference similarity score (e.g., a score corresponding to 95% similarity), the electronic device may evaluate the first response as correct. For example, if the similarity score of the first similarity information is less than the reference similarity score, the electronic device may evaluate the first response as not correct. In this way, if the similarity between the response to the hint and the correct answer is equal to or greater than the reference similarity, the electronic device may evaluate the response as correct. In other words, even if the response and the correct answer do not perfectly match, the electronic device may evaluate the response as correct if they are similar to a certain level or higher.

[0163] According to one embodiment, the correct answer evaluation information (642) for the first response may include evaluation result information for the first response (hereinafter, referred to as the first evaluation result information). The first evaluation result information may be set to either a first value (e.g., 0) indicating that the first response is evaluated as correct or a second value (e.g., 1) indicating that the first response is evaluated as not correct. When the correct answer evaluation information (642) includes the first evaluation result information, the electronic device may directly determine whether the first response is correct based on the first evaluation result information without the separate similarity score comparison judgment process described above. For example, when the value of the first evaluation result information is set to the first value (e.g., 0), the electronic device may identify the first response as correct. For example, when the value of the first evaluation result information is set to the second value (e.g., 1), the electronic device may identify the first response as not correct.

[0164] According to one embodiment, in operation 560, the electronic device may generate second hint information (641b) associated with the correct answer information (633) using a first generative AI model (e.g., a language AI model) based on the evaluation result, correct answer information (633), information related to the user's behavioral pattern (632), and / or first feedback information, and provide the second hint information (641b).

[0165] According to one embodiment, the first feedback information may include a first response and / or first hint information (641a). For example, compared to operation 530 for generating the first hint information (641a), which is the initial hint information, operation 560 for generating the second hint information (641b), which is additional hint information, may further use feedback information including previous hint information (e.g., the first hint information (641a)) and / or a response to the previous hint information (e.g., the first response) in addition to the correct answer information (633) and / or information related to the user's behavior pattern (632) to generate the second hint information (641b). In this way, since a response including the previous hint information and / or information remembered by the user through the previous hint information is used to generate additional hint information as feedback information, the generated additional hint information may additionally provide information that helps the user better associate the correct answer information compared to the previous hint information. Electronic devices can help users recover and strengthen their memories by supporting associative learning through hints provided sequentially in this manner.

[0166] In one embodiment, the second hint information (641b) may include second information that helps to recall the correct answer information (633). In one embodiment, the second hint information (641b) may be different from the first hint information (641a).

[0167] In one embodiment, the second hint information (641b) may include more detailed information related to the correct answer information (633) than the first hint information (641a). Through the second hint information (641b) provided in this manner, the user can better associate the correct answer information compared to when the first hint information (641b) is provided.

[0168] In one embodiment, the second hint information (641b) may have a higher similarity to the correct answer information (633) than the first hint information (641a). Through the second hint information (641b) provided in this manner, the user can better associate the correct answer information compared to when the first hint information (641b) is provided.

[0169] In one embodiment, the second hint information (641b) may include more personalized information related to the user than the first hint information (641a). For example, since each user has different tendencies, the method or characteristics of deriving the correct answer from the hint may also differ. For example, the first user may be better able to derive the correct answer through hints associated with temporal elements (e.g., time, frequency) than through hints associated with spatial elements (e.g., place, location). In this case, the first hint information (641a) for the first user may be generated using the first behavioral element associated with the spatial element included in the information related to the user's behavioral pattern (632), and the second hint information (641b) for the first user may be generated using the second behavioral element associated with the temporal element. For example, the second user may be better able to derive the correct answer through hints associated with environmental / situational elements (e.g., people around, interactions) than through hints associated with temporal elements. In this case, the first hint information (641a) for the second user may be generated using a second behavioral element associated with a temporal element included in information (632) related to the user's behavioral pattern, and the second hint information (641b) for the second user may be generated using a third behavioral element associated with a situational element. Through the second hint information (641b) provided in this personalized manner, the user may be able to better associate the correct answer information compared to when the first hint information (641b) is provided.

[0170] According to one embodiment, the information (632) related to the user's behavior pattern may include a plurality of pieces of information, and second data including at least one piece of information among the plurality of pieces of information may be used to generate second hint information (641b), and first data including at least one piece of information among the plurality of pieces of information may be used to generate first hint information (641a). In this case, the second data and the first data may be different. For example, the number of behavioral elements included in the information (632) related to the user's behavior pattern used to generate the second hint information (641b) may be greater than the number of behavioral elements included in the information (632) related to the user's behavior pattern used to generate the first hint information (641a). For example, assume that the information (632) related to the user's behavior pattern includes a first behavioral element associated with a temporal element and a second behavioral element associated with a spatial element. At this time, the second hint information (641b) may be generated using both the first action element and the second action element, and the first hint information (641a) may be generated using only the first action element. Through this, a greater amount of the user's behavioral patterns and / or daily habits may be reflected in the second hint information (641b) compared to the first hint information (641a). Through the second hint information (641b) provided in this manner, the user can better associate the correct answer information compared to when the first hint information (641b) is provided. Examples of the first hint information (641a) and the second hint information (641b) are described below with reference to FIGS. 12b and 13b.

[0171] According to one embodiment, the language AI model used to generate the second hint information (641b) may be the same as the language AI model used to generate the first hint information (641a).

[0172] In one embodiment, if the first response is identified as correct based on the correct answer evaluation information (642), the electronic device may provide (e.g., display, projection, or audio output) information indicating that the first response is correct and / or correct answer information. In this case, the electronic device no longer needs to generate and provide additional hint information to the user.

[0173] According to one embodiment, if the first response is identified as not being correct based on the correct answer evaluation information (642), the electronic device may generate second hint information (641b) associated with the correct answer information (633) based on the correct answer information (633), information related to the behavior pattern (632), and / or the first feedback information, and provide the second hint information (641b). As described above, the first feedback information may include the first response and / or the first hint information (641a).

[0174] According to one embodiment, the electronic device may use the language module (640) to generate second hint information (641b) associated with the correct answer information (633), information related to the user's behavior pattern (632), and / or first feedback information. For example, the language module (640) may input the correct answer information (633), information related to the user's behavior pattern (632), and / or first feedback information into a first language AI model (e.g., a first AI model included in a first generative AI model (e.g., LLM)), and generate second hint information (641b) associated with the correct answer information (633) from the first language AI model. According to one embodiment, the language AI model used to generate the first hint information (641a) and the language AI model used to generate the second hint information (641b) may be the same AI model.

[0175] According to one embodiment, the electronic device can visually provide the second hint information (641b) via a display (e.g., the display (331) of FIGS. 3A and 3B) or a projector (e.g., the projector (332) of FIGS. 3A and 3B). For example, the electronic device can display the second hint information (641b) via a display. For example, the electronic device can project the second hint information (641b) onto a projection surface (e.g., a floor, a wall, a screen) via a projector.

[0176] In one embodiment, the electronic device may audibly provide the second hint information (641b) via a speaker (e.g., speaker (362) of FIGS. 3A and 3B ). For example, the electronic device may output a sound corresponding to the second hint information (641b) via the speaker.

[0177] According to one embodiment, after the second hint information (641b) is provided, the electronic device may perform operations corresponding to operations 540 to 560 for the second hint information (641b) to generate and provide next hint information (e.g., third hint information). For example, as in operations 540 and 550, the electronic device may obtain a second user input (612b) including a second response to the second hint information (641b) and evaluate the second response based on the correct answer information (633). For example, as in operation 560, the electronic device may generate next hint information associated with the correct answer information (633) using the first generative AI model based on the correct answer information (633), the information related to the behavior pattern (632) and / or the second feedback information, and provide the next hint information, according to the evaluation result. The second feedback information may include the second response and / or the second hint information (641b). The operations corresponding to operations 540 to 560 for generating and providing the following hint information may be repeatedly performed until a specified condition is satisfied (e.g., until the response to the current hint information is evaluated or identified as being correct).

[0178] FIG. 8 is a flowchart illustrating a method by which an electronic device generates scene data and information related to a user's behavioral pattern, according to one embodiment of the present disclosure.

[0179] FIG. 9A is a diagram illustrating scene data according to one embodiment of the present disclosure.

[0180] FIG. 9b is a diagram illustrating information related to a user's behavior pattern according to one embodiment of the present disclosure.

[0181] FIG. 10 is a flowchart illustrating a method for an electronic device to generate scene data according to one embodiment of the present disclosure.

[0182] FIG. 11 is a flowchart illustrating a method for an electronic device to generate information related to a user's behavioral pattern, according to one embodiment of the present disclosure.

[0183] The components and operations of the components described with reference to FIGS. 8 to 11 may be partially or entirely identical to the components and operations of the components described with reference to FIGS. 1 to 7.

[0184] For convenience of explanation, in the embodiments of FIGS. 8 to 11, the electronic device is described as an example of a mobile electronic device (300) of FIGS. 2, 3a, and 3b. However, the embodiments are not limited thereto, and the description of the embodiments of FIGS. 5 to 7 may also be applied to other types of electronic devices (e.g., the electronic device (101) of FIG. 1a, the server (108) of FIG. 1a, or the wearable device (400) of FIGS. 2, 4a to 4c). In the following embodiments, the operation of the electronic device may be understood as being performed by at least one processor included in the electronic device (e.g., the processor (390) of FIGS. 3a and 3b).

[0185] In the following examples, the operations may be performed sequentially, but are not necessarily sequential. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0186] Referring to FIGS. 8 to 11, according to one embodiment, in operation 810, the electronic device may generate at least one scene data (e.g., scene data 631 of FIG. 6) using a second generative AI model (e.g., LMM) based on sensing data (e.g., first sensing data (621) and / or second sensing data (622) of FIG. 6) associated with a user (e.g., user (U) of FIG. 2). For example, as illustrated in FIG. 10, the electronic device may obtain sensing data associated with a user (1010), extract feature information for each modality data included in the sensing data (1020), and generate at least one scene data (1030) based on the feature information. Actions 1010 to 1030 can be performed using a first AI model included in a second generative AI model (e.g., a first multimodal AI model included in a multimodal module (630) of FIG. 6).

[0187] In one embodiment, the electronic device may generate new scene data whenever a scene data generation condition is satisfied. For example, the electronic device may generate new scene data whenever session sensing data is generated or acquired.

[0188] According to one embodiment, an electronic device may monitor a user to obtain sensing data, and segment and sessionize the sensing data to obtain at least one session sensing data. Segmentation may include dividing continuous sensing data into meaningful units. Sessionization may include grouping the segmented sensing data into session units.

[0189] According to one embodiment, each session sensing data may include sensing data grouped into a corresponding session. As described above, sessions may be distinguished based on, for example, a specific user activity, a specific location, a specific event, and / or a specific time. For example, sensing data associated with a specific user activity (e.g., a phone call, waking up in the morning, eating breakfast, resting) may be grouped into a single session. For example, sensing data for the time a user stays in a specific location (e.g., a room) may be grouped into a single session. For example, sensing data associated with a specific event may be grouped into a single session. The sensing data included in each session sensing data may include at least a portion of continuous sensing data acquired by monitoring the user.

[0190] According to one embodiment, the electronic device may generate scene data, based on session sensing data, that includes information (e.g., summary information) about at least one scene associated with a session of the session sensing data.

[0191] According to one embodiment, in operation 820, the electronic device may generate information related to a user's behavior pattern (e.g., information related to a user's behavior pattern (632) of FIG. 6) based on at least one scene data. For example, as illustrated in FIG. 11, the electronic device collects scene data (1110), determines (1120) whether scene data or a specified number or more of scene data is collected during a specified period (e.g., one day, one week, one month), and if scene data or a specified number or more of scene data is collected during the specified period, the electronic device generates (1130) information related to the user's behavior pattern using a third AI model included in a second generative AI model based on the collected scene data set (e.g., a third multimodal AI model included in a multimodal module (630) of FIG. 6), and if scene data or a specified number or more of scene data is not collected during the specified period, the electronic device may continue to collect scene data (1110) again. The scene data set collected during the specified period or a specified number or more of scene data may include at least one scene data.

[0192] According to one embodiment, FIG. 9A illustrates scene data generated, for example, after a user wakes up in the morning.

[0193] For example, part (a) of FIG. 9A illustrates first scene data (901) associated with a user's morning wake-up activity. The first scene data (901) may be generated based on first session sensing data (e.g., image data or biometric data associated with the morning wake-up activity) grouped into a first session associated with the user's morning wake-up activity.

[0194] For example, part (b) of FIG. 9A illustrates second scene data (902) associated with a user's breakfast preparation activity. The second scene data (902) may be generated based on second session sensing data (e.g., location data, motion data, voice data associated with the breakfast preparation activity) grouped into a second session associated with the user's breakfast preparation activity.

[0195] For example, part (c) of FIG. 9A illustrates third scene data (903) associated with a user's rest activity. The third scene data (903) may be generated based on third session sensing data (e.g., image data, biometric data, sound data related to the rest activity) grouped into a third session associated with the user's rest activity.

[0196] According to one embodiment, the electronic device can obtain first session sensing data, second session sensing data, and third session sensing data by monitoring the same user, dividing the sensing data collected during a specified period, and sessionizing them for each session.

[0197] According to one embodiment, FIG. 9b illustrates information related to a user's behavioral pattern generated based on, for example, the scene data of FIG. 9a.

[0198] For example, part (a) of FIG. 9B illustrates first behavior pattern-related information (911) that includes summary information about a user's first behavior pattern or habit (e.g., morning routine). The first behavior pattern-related information (911) may be generated based on a scene data set including first scene data (901) and second scene data (902). The first behavior pattern-related information (911) may include a first behavior element associated with a time element (e.g., 7:00 AM) and / or a second behavior element associated with a location element (e.g., kitchen). The first behavior pattern-related information (911) may be used to assess whether a user maintains a regular lifestyle.

[0199] For example, part (b) of FIG. 9B illustrates second behavior pattern-related information (912) including summary information about a user's second behavior pattern or habit (e.g., balance between activity and rest). The second behavior pattern-related information (912) may be generated based on a scene data set collected over a specified period (e.g., one week) including first scene data (901), second scene data (902), and third scene data (903). The second behavior pattern-related information (912) may include at least one first behavior element associated with a time element (e.g., 3 hours, 32 hours, remaining time) and / or at least one second behavior element associated with a location element (e.g., home). The second behavior pattern-related information (912) may be used to evaluate whether the user's daily activities are balanced.

[0200] FIG. 12A is a diagram illustrating information that a user wants to remember, according to one embodiment of the present disclosure.

[0201] FIG. 12b is a diagram illustrating hint information associated with correct answer information corresponding to information that the user of FIG. 12a wants to remember, according to one embodiment of the present disclosure.

[0202] The components and operations of the components described with reference to FIGS. 12a and 12b may be partially or entirely identical to the components and operations of the components described with reference to FIGS. 1 to 11.

[0203] For convenience of explanation, in the embodiments of FIGS. 12a and 12b, the electronic device is described as an example of a mobile electronic device (300) of FIGS. 2, 3a, and 3b. However, the embodiment is not limited thereto, and the description of the embodiments of FIGS. 12a and 12b may also be applied to other types of electronic devices (e.g., the electronic device (101) of FIG. 1a, the server (108) of FIG. 1a, or the wearable device (400) of FIGS. 2, 4a to 4c). In the following embodiments, the operation of the electronic device may be understood as being performed by at least one processor included in the electronic device (e.g., the processor (390) of FIGS. 3a and 3b).

[0204] In the embodiments of FIGS. 12A and 12B , a user (U1) may be in a situation where he or she had a phone conversation with his or her son (U2) yesterday, but cannot quite recall the content of the conversation the next day. In this situation, as illustrated in FIG. 12A , the information (1210) that the user (U1) wishes to remember may correspond to the conversation he or she had with his or her son yesterday. The information (1210) that the user (U1) wishes to remember may be provided to the electronic device (300) via a first user input (e.g., the first user voice input (611) of FIG. 6 ). As illustrated in FIG. 12b, the electronic device (300) may provide at least one hint information (1221, 1222, 1223) (e.g., first hint information (641a), second hint information (641b) of FIG. 6) associated with correct answer information (e.g., correct answer information (633) of FIG. 6) for information (1210) that the user wishes to remember. According to one embodiment, the at least one hint information (1221, 1222, 1223) may be generated without using the user's behavior pattern information (e.g., the user's behavior pattern information (632) of FIG. 6). For example, the at least one hint information (1221, 1222, 1223) may be generated in association with a topic according to the chronological order of the conversation, respectively.

[0205] According to one embodiment, the electronic device (300) may provide first hint information (1221) corresponding to an initial hint (e.g., first hint information (641a) of FIG. 6). The first hint information (1221) may include information (e.g., "I spoke with my son on the phone last night. What did he say about work at the beginning of the call?") ​​that helps the user (U1) remember a topic related to the son's work (e.g., information about each other's recent activities) mentioned by the son (U2) at the beginning of the call.

[0206] According to one embodiment, the electronic device (300) may provide second hint information (1222) corresponding to an additional hint (e.g., second hint information (641b) of FIG. 6). The second hint information (1222) may include information that helps the user (U1) to more clearly remember the health-related conversation he had with his son (U2) during the call by specifically addressing the health-related conversation content among the conversation contents (e.g., “The health status of your family was also mentioned during the call. What questions did your son ask Mr. OOO regarding the health status of his family members?”).

[0207] In one embodiment, the electronic device (300) may provide third hint information (1223) corresponding to the final hint. The third hint information (1223) may provide information at the end of the call that helps the user (U1) recall details about the family gathering and travel plans suggested by the son (U2), such as, "What is the upcoming event that the son mentioned at the end of the conversation and what are the plans for it? Remember what you decided to do about this event?", thereby helping the user (U1) to more accurately reconstruct the overall conversation.

[0208] As described above, electronic devices can help users recall important points of a conversation by sequentially providing hints that remind them of the topics of the conversation during the call in chronological order.

[0209] FIG. 13A is a diagram illustrating information that a user wants to remember, according to one embodiment of the present disclosure.

[0210] FIG. 13b is a diagram illustrating hint information associated with correct answer information corresponding to information that the user of FIG. 13a wants to remember, according to one embodiment of the present disclosure.

[0211] The components and operations of the components described with reference to FIGS. 13a and 13b may be partially or entirely identical to the components and operations of the components described with reference to FIGS. 1 to 12.

[0212] For convenience of explanation, in the embodiments of FIGS. 13a and 13b, the electronic device is described as an example of a mobile electronic device (300) of FIGS. 2, 3a, and 3b. However, the embodiment is not limited thereto, and the description of the embodiments of FIGS. 13a and 13b may also be applied to other types of electronic devices (e.g., the electronic device (101) of FIG. 1a, the server (108) of FIG. 1a, or the wearable device (400) of FIGS. 2, 4a to 4c). In the following embodiments, the operation of the electronic device may be understood as being performed by at least one processor included in the electronic device (e.g., the processor (390) of FIGS. 3a and 3b).

[0213] In the embodiments of FIGS. 13A and 13B , a user (U1) suffering from cognitive impairment may be in a situation where they are trying to remember what they did after eating breakfast today, but are having difficulty remembering. In this situation, as illustrated in FIG. 13A , the information (1310) that the user (U1) wishes to remember may correspond to the activities they did after eating breakfast today. The information (1310) that the user (U1) wishes to remember may be provided to the electronic device (300) via a first user input (e.g., the first user voice input (611) of FIG. 6 ). As illustrated in FIG. 13b, the electronic device (300) can provide at least one hint information (1321, 1322, 1323) (e.g., first hint information (641a), second hint information (641b) of FIG. 6) associated with correct answer information (e.g., correct answer information (633) of FIG. 6) for information (1310) that the user wants to remember.

[0214] According to one embodiment, the electronic device (300) may generate at least one hint information (1321, 1322, 1323) associated with correct answer information by using information related to the user's behavioral pattern (e.g., the user's behavioral pattern information (632) of FIG. 6). In this way, the electronic device (300) generates hint information by reflecting information about the user's behavioral pattern (e.g., daily habits), thereby providing hint information related to the context of daily life, thereby enabling the user to recall memories more clearly.

[0215] According to one embodiment, the electronic device (300) may provide first hint information (1321) corresponding to an initial hint (e.g., first hint information (641a) of FIG. 6). The first hint information (1321) may include information that helps to stimulate memory by reminding the user (U1) of the daily activities and environments they perform (e.g., “This morning, what did you do while sitting by the living room window? Think about what you usually do at this time.”).

[0216] According to one embodiment, the electronic device (300) may provide second hint information (1322) corresponding to an additional hint (e.g., second hint information (641b) of FIG. 6). The second hint information (1322) may focus on a more specific activity, such as drinking tea, compared to the first hint information (1331), and may include information that helps the user (U1) remember more easily (e.g., “What did Mr. OOO drink while sitting by the window? This is a drink that Mr. OOO likes to drink in the morning.”).

[0217] According to one embodiment, the electronic device (300) may provide third hint information (1323) corresponding to the last hint. The third hint information (1323) may include information that helps the user (U1) remember the moment more specifically by providing hints about details related to the surrounding environment (e.g., "What did you see outside the window while drinking tea? This scenery makes you feel peaceful and stable every morning").

[0218] As described above, the electronic device (300) generates hint information by reflecting information about the user's behavioral patterns (e.g., daily habits), thereby providing hint information relevant to the context of daily life for associative learning. This can help restore and strengthen the user's memory.

[0219] FIG. 14 is a flowchart illustrating a method for an electronic device to provide associative learning according to one embodiment of the present disclosure.

[0220] The components and operations of the components described with reference to FIG. 14 may be partially or entirely identical to the components and operations of the components described with reference to FIGS. 1 to 13. In the embodiments below, the respective operations may be performed sequentially, but are not necessarily performed sequentially. For example, the order of the respective operations may be changed, and at least two operations may be performed in parallel. In the embodiments below, the operations of the electronic device may be understood as being performed by at least one processor included in the electronic device.

[0221] In the embodiment of FIG. 14, the first user input to the third user input refer to the first input, the second input, and the third input of the same user (e.g., the user (U) of FIG. 2).

[0222] Referring to FIG. 14, according to one embodiment, in operation 1401, an electronic device (e.g., an electronic device (101) of FIG. 1A, a server (108) of FIG. 1A, a mobile electronic device (300) of FIGS. 2, 3A, and 3B, or a wearable electronic device (400) of FIGS. 2, 4A to 4C) may obtain first sensing data (e.g., first sensing data (621) of FIG. 6) associated with a user (e.g., a user (U) of FIG. 2). According to one embodiment, in operation 1402, the electronic device may obtain (or receive) second sensing data (e.g., second sensing data (622) of FIG. 6) associated with a user (e.g., a user (U) of FIG. 2) from an external electronic device (e.g., a wearable electronic device (400) of FIGS. 2, 4A to 4C). Actions 1401 and 1402 may be performed simultaneously or in parallel, or one action may be performed before the other.

[0223] According to one embodiment, in operation 1403, the electronic device may generate at least one scene data (e.g., scene data (631) of FIG. 6) associated with a user activity based on the first sensing data and the second sensing data. For example, the electronic device may generate the scene data using a first AI model trained to generate the scene data based on the first sensing data and the second sensing data (or multimodal feature information extracted from the first sensing data and the second sensing data). According to one embodiment, the first AI model may be an AI model included in a second generative AI model (e.g., LMM). The first AI model may be, for example, a first multimodal AI model included in the multimodal module (630) of FIG. 6.

[0224] According to one embodiment, the electronic device may transmit at least a portion of the generated scene data to a server (e.g., server 650 of FIG. 6). Since the amount of generated scene data may continuously increase over time, if the memory of the electronic device is insufficient, the electronic device may transmit at least a portion of the generated scene data to the server so that the scene data can be managed by the server. When transmitting the scene data to the server, the electronic device may encrypt and compress the transmitted scene data to protect personal information and transmit it. The electronic device may request scene data managed by the server from the server when necessary, and receive and use the requested scene data from the server.

[0225] According to one embodiment, in operation 1404, the electronic device may generate information related to a user's behavioral pattern (e.g., information related to a user's behavioral pattern (632) of FIG. 6) based on at least one scene data. For example, the electronic device may generate information related to the user's behavioral pattern using a second AI model trained to generate information related to the user's behavioral pattern based on the scene data. According to one embodiment, the second AI model may be an AI model included in a second generative AI model (e.g., LMM). The second AI model may be, for example, a second multimodal AI model included in a multimodal module (630) of FIG. 6. The information related to the user's behavioral pattern may include information summarizing the user's behavioral pattern over a certain period of time (e.g., one day, one week, one month). Through the information related to the user's behavioral pattern, it may be possible to determine what behavioral pattern or daily habit the user has.

[0226] According to one embodiment, in operation 1405, the electronic device may identify whether a first user input (e.g., the first user voice input (611) of FIG. 6) is received. The first user input may be, for example, a user input requesting information that the user wishes to remember. If it is identified that the first user input is received, operation 1406 may be performed. If it is identified that the first user input is not received, operation 1401 may be performed again.

[0227] According to one embodiment, in operation 1406, the electronic device may generate correct answer information (e.g., correct answer information (633) of FIG. 6) for information corresponding to the first user input based on at least one scene data and / or information corresponding to the first user input. For example, the electronic device may generate information related to the user's behavior pattern using a third AI model trained to generate correct answer information based on the scene data and / or information corresponding to the first user input. According to one embodiment, the third AI model may be an AI model included in a second generative AI model (e.g., LMM). The third AI model may be, for example, a third multimodal AI model included in a multimodal module (630) of FIG. 6. The information corresponding to the user input may include, for example, information that the user wants to remember.

[0228] According to one embodiment, in operation 1407, the electronic device may generate and provide (e.g., display, project, sound output) first hint information (e.g., first hint information 641a of FIG. 6) associated with correct answer information based on information related to the user's behavior pattern and / or correct answer information. For example, the electronic device may generate first hint information associated with correct answer information using a fourth AI model trained to generate hint information based on information related to the user's behavior pattern and / or correct answer information. According to one embodiment, the fourth AI model may be an AI model included in a first generative AI model (e.g., LLM). The fourth AI model may be, for example, a first language AI model included in a language module (640) of FIG. 6.

[0229] According to one embodiment, in operation 1408, the electronic device may identify whether a second user input (e.g., the second user voice input (612b) of FIG. 6) associated with the first hint information is received. The second user input may be, for example, a user input requesting confirmation of information the user has recalled through the first hint information. If it is identified that the second user input is received, operation 1409 may be performed. If it is identified that the second user input is not received, operation 1413 may be performed.

[0230] According to one embodiment, in operation 1409, the electronic device may obtain (or generate) similarity score information based on correct answer information and / or information corresponding to a second user input (e.g., a first response to the first hint information). For example, the electronic device may generate a similarity score using a fifth AI model trained to generate similarity score information based on correct answer information and / or information corresponding to a second user input. According to one embodiment, the fifth AI model may be an AI model included in a first generative AI model (e.g., LLM). The fifth AI model may be, for example, a second language AI model included in a language module (640) of FIG. 6. The first response to the first hint information may include, for example, information remembered by the user through the first hint information. The similarity score information may indicate a similarity score between the first response and the correct answer information.

[0231] According to one embodiment, in operation 1410, the electronic device may determine whether the similarity score is less than a reference similarity score. If the similarity score is less than the reference similarity score, operation 1411 may be performed. If the similarity score is greater than or equal to the reference similarity score, operation 1413 may be performed.

[0232] According to one embodiment, in operation 1411, the electronic device may generate and provide (e.g., display, project, or output sound) second hint information (e.g., second hint information 641b of FIG. 6) associated with the correct answer information based on information related to the user's behavioral pattern, correct answer information, and / or first feedback information. For example, the electronic device may generate second hint information associated with the correct answer information using a sixth AI model trained to generate hint information based on information related to the user's behavioral pattern, correct answer information, and / or first feedback information. The first feedback information may include a first response and / or first hint information. According to one embodiment, the sixth AI model may be an AI model included in a first generative AI model (e.g., LLM). The sixth AI model used to generate the second hint information may be the same AI model as the fourth AI model used to generate the first hint information. For example, the sixth AI model may be a first language AI model included in a language module (640) of FIG. 6.

[0233] According to one embodiment, at operation 1412, the electronic device may determine whether a third user input requesting direct provision of correct information is received. If it is determined that the third user input has been received, operation 1413 may be performed. If it is determined that the third user input has not been received, operation 1408 may be performed again.

[0234] According to one embodiment, in operation 1413, the electronic device may directly provide (e.g., display, project, or output sound) the correct answer information. For example, if no second user input related to the first hint information is received (e.g., if the first hint information does not request confirmation of the information the user has recalled), if the similarity score is greater than or equal to a reference similarity score (e.g., if the information the user has recalled is close to the correct answer), or if a third user input requesting direct provision of the correct answer information is received, the electronic device may directly provide the correct answer information. In this case, the electronic device may also provide a notification or feedback indicating that the user has guessed the correct answer.

[0235] According to one embodiment of the present disclosure, an electronic device can support associative learning.

[0236] According to one embodiment of the present disclosure, an electronic device can provide hint information associated with the correct answer to information a user wishes to memorize. This allows the electronic device to support associative learning.

[0237] According to one embodiment of the present disclosure, an electronic device can generate hint information using contextual information associated with the user (e.g., information related to the user's behavioral patterns). The hint information generated in this way reflects contextual information (e.g., the user's behavioral patterns or daily habits), thereby helping the user better remember the correct answer. This allows the electronic device to support associative learning with greater effectiveness.

[0238] According to one embodiment of the present disclosure, an electronic device can evaluate information recalled by a user through hint information, and based on the evaluation results, provide additional hint information associated with the correct answer using feedback information. The additional hint information can be generated not only by information related to the user's behavioral patterns, but also by further utilizing the information recalled by the user as feedback information. The additional hint information generated in this manner can include information that helps the user better recall the correct answer compared to previous hints. By providing such sequential hints, the electronic device can enhance the effectiveness of supported associative learning.

[0239] According to one embodiment of the present disclosure, an electronic device (e.g., a mobile electronic device (300) of FIGS. 2, 3A, and 3B) supporting associative learning is provided. The electronic device may include a memory (e.g., a memory (380) of FIG. 3B) including at least one storage medium for storing instructions; and at least one processor (e.g., a processor (390) of FIG. 3B) including a processing circuit. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to perform at least one operation. The at least one operation may include an operation of generating correct answer information (e.g., correct answer information (633) of FIG. 6) for information corresponding to a first user input (e.g., a first user voice input (611) of FIG. 6). The at least one operation may include an operation of generating first hint information (e.g., first hint information (641a) of FIG. 6) associated with the correct answer information and information related to a user's behavioral pattern (e.g., information related to a user's behavioral pattern (632) of FIG. 6) using a first generative AI (artificial intelligence) model (e.g., a language AI model included in a language module (640) of FIG. 6) based on the correct answer information and information related to a user's behavioral pattern (e.g., information related to a user's behavioral pattern (632) of FIG. 6), and providing the first hint information. The at least one operation may include an operation of obtaining a second user input (e.g., second user input (612a) of FIG. 6) including a first response (answer) to the first hint information. The at least one operation may include an operation of evaluating the first response based on the correct answer information.The at least one action may include an action of generating second hint information (e.g., second hint information (641b) of FIG. 6) associated with the correct answer information using the first generative AI model based on the correct answer information, information related to the behavioral pattern, and first feedback information, according to a result of the evaluation, and providing the second hint information. The first feedback information may include at least one of the first response or the first hint information.

[0240] According to one embodiment of the present disclosure, a method of operating an electronic device is provided. The method of operating the electronic device may include at least one operation. The at least one operation may include an operation of generating correct answer information (e.g., correct answer information (633) of FIG. 6) for information corresponding to a first user input (e.g., a first user voice input (611) of FIG. 6). The at least one operation may include an operation of generating first hint information (e.g., first hint information (641a) of FIG. 6) associated with the correct answer information using a first generative artificial intelligence (AI) model (e.g., a language AI model included in a language module (640) of FIG. 6) based on the correct answer information and information related to a user's behavior pattern (e.g., information related to a user's behavior pattern (632) of FIG. 6), and providing the first hint information. The at least one operation may include an operation of obtaining a second user input (e.g., the second user input (612a) of FIG. 6) including a first response to the first hint information. The at least one operation may include an operation of evaluating the first response based on the correct answer information. The at least one operation may include an operation of generating second hint information (e.g., the second hint information (641b) of FIG. 6) associated with the correct answer information using the first generative AI model based on the correct answer information, information related to the behavior pattern, and first feedback information, according to a result of the evaluation, and providing the second hint information. The first feedback information may include at least one of the first response or the first hint information.

[0241] According to one embodiment, the electronic device may include at least one sensor (e.g., the sensor module (340) of FIG. 3B) and a communication circuit (e.g., the communication module (320) of FIG. 3B). The at least one operation may include an operation of generating at least one scene data (e.g., the scene data (631) of FIG. 6) associated with an action of the user using a second generative AI model (e.g., a multimodal AI model included in the multimodal module (630) of FIG. 6) based on at least one of first sensing data (e.g., the first sensing data (621) of FIG. 6) associated with the user acquired through the at least one sensor or second sensing data (e.g., the second sensing data (622) of FIG. 6) associated with the user received from an external electronic device (e.g., the wearable electronic device (400) of FIGS. 2, 4A to 4C) through the communication circuit. Information related to the user's behavior pattern is generated based on the at least one scene data, and the first sensing data may include at least one of image data, voice data, or movement data of the user, and the second sensing data may include at least one of voice data, movement data, or biometric data of the user.

[0242] According to one embodiment, the correct answer information may be generated using the second generative AI model based on the at least one scene data and information corresponding to the first user input.

[0243] According to one embodiment, the information related to the behavior pattern includes at least one behavior element, and the at least one behavior element includes at least one of a first behavior element associated with a temporal element, a second behavior element associated with a spatial element, or a third behavior element associated with a situational element, and the number of the behavior elements used to generate the second hint information may be greater than the number of the behavior elements used to generate the first hint information.

[0244] According to one embodiment, the second hint information may include more detailed information related to the correct answer information than the first hint information.

[0245] According to one embodiment, the second hint information may include more personalized information associated with the user than the first hint information.

[0246] According to one embodiment, the result of the evaluation may include information indicating a similarity score between the first response and the correct answer information (e.g., correct answer evaluation information (642) of FIG. 6). The at least one operation may include an operation of generating the second hint information based on an identification that the similarity score is less than a reference similarity score, and an operation of providing the correct answer information based on an identification that the similarity score is greater than or equal to a reference similarity score.

[0247] According to one embodiment, the at least one operation may include, prior to generating the first hint information, an operation of obtaining information related to a behavioral pattern of the user as source data used to generate the first hint information based on at least one of information corresponding to the first user input or the correct answer information.

[0248] According to one embodiment, the at least one operation may include obtaining a third user input directly requesting the correct answer information, and providing the correct answer information in response to the third user input.

[0249] According to one embodiment, the information corresponding to the first user input may include content that the user wishes to remember, and the first response may include content that the user has remembered through the first hint information.

[0250] The embodiments of this document and the terminology used herein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0251] The term "module" used in the embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0252] One embodiment of the present document may be implemented as software (e.g., a program (140) of FIG. 1) including one or more instructions stored in a storage medium (e.g., an internal memory (136) of FIG. 1 or an external memory (138) of FIG. 1) readable by a machine (e.g., an electronic device (101) of FIG. 1). For example, a processor (e.g., a processor (120) of FIG. 1) of the machine (e.g., an electronic device (101) of FIG. 1) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0253] According to one embodiment, the method according to one embodiment disclosed in this document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play Store™) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0254] According to one embodiment, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and arranged in other components. According to one embodiment, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to one embodiment, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In electronic devices, a memory comprising at least one storage medium for storing instructions; and comprising at least one processor comprising a processing circuit; The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: Generate correct answer information for information corresponding to the first user input, Based on the above correct answer information and information related to the user's behavior pattern, first hint information related to the correct answer information is generated using a first generative AI (artificial intelligence) model, and the first hint information is provided. Obtaining a second user input including a first response to the first hint information, Based on the above correct answer information, evaluate the first response, Based on the results of the above evaluation, the first generative AI model is used to generate second hint information related to the correct answer information, information related to the behavior pattern, and first feedback information, and the second hint information is provided. An electronic device, wherein the first feedback information includes at least one of the first response or the first hint information.

2. In the first paragraph, at least one sensor and communication circuit are included, The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: Generating at least one scene data associated with the user's behavior using a second generative AI model based on at least one of first sensing data associated with the user acquired through the at least one sensor or second sensing data associated with the user received from an external electronic device through the communication circuit, Information related to the user's behavioral pattern is generated based on at least one scene data, An electronic device, wherein the first sensing data includes at least one of image data, voice data, or movement data of the user, and the second sensing data includes at least one of voice data, movement data, or biometric data of the user.

3. In paragraph 2, An electronic device wherein the above correct answer information is generated using the second generative AI model based on the at least one scene data and the information corresponding to the first user input.

4. In any one of paragraphs 1 to 3, The information related to the above behavior pattern includes at least one behavior element, and the at least one behavior element includes at least one of a first behavior element associated with a temporal element, a second behavior element associated with a spatial element, or a third behavior element associated with a situational element. An electronic device wherein the number of the behavioral elements used to generate the second hint information is greater than the number of the behavioral elements used to generate the first hint information.

5. In any one of paragraphs 1 to 3, An electronic device wherein the second hint information includes more detailed information related to the correct answer information than the first hint information.

6. In any one of paragraphs 1 to 3, An electronic device wherein the second hint information includes more personalized information associated with the user than the first hint information.

7. In any one of paragraphs 1 to 6, The result of the above evaluation includes information indicating a similarity score between the first response and the correct answer information, The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: Based on the identification that the above similarity score is less than the reference similarity score, the second hint information is generated, An electronic device that causes the correct answer information to be provided based on the identification that the similarity score is greater than or equal to a reference similarity score.

8. In any one of paragraphs 1 to 7, The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: An electronic device that causes information related to a behavioral pattern of the user to be acquired as source data used for generating the first hint information, based on at least one of the information corresponding to the first user input or the correct answer information, prior to generating the first hint information.

9. In any one of paragraphs 1 to 8, The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: Obtaining a third user input requesting the above correct answer information, An electronic device that causes the third user input to provide the correct answer information.

10. In any one of paragraphs 1 to 9, The information corresponding to the first user input includes content that the user wants to remember, An electronic device wherein the first response includes content remembered by the user through the first hint information.

11. In the method of an electronic device, An action to generate correct answer information for information corresponding to a first user input; An operation of generating first hint information related to the correct answer information and information related to the user's behavior pattern using a first generative AI (artificial intelligence) model and providing the first hint information; An operation of obtaining a second user input including a first response (answer) to the first hint information; An operation of evaluating the first response based on the above correct answer information; and Based on the results of the above evaluation, an operation is included for generating second hint information related to the correct answer information using the first generative AI model based on the correct answer information, information related to the behavior pattern, and first feedback information, and providing the second hint information. A method wherein the first feedback information includes at least one of the first response or the first hint information.

12. In the 11th paragraph, the electronic device includes at least one sensor and communication circuit, and the method: An operation of generating at least one scene data associated with the user's behavior using a second generative AI model based on at least one of first sensing data associated with the user acquired through the at least one sensor or second sensing data associated with the user received from an external electronic device through the communication circuit, Information related to the user's behavioral pattern is generated based on at least one scene data, A method wherein the first sensing data includes at least one of image data, voice data, or movement data of the user, and the second sensing data includes at least one of voice data, movement data, or biometric data of the user.

13. In paragraph 12, A method wherein the above correct answer information is generated using the second generative AI model based on the at least one scene data and the information corresponding to the first user input.

14. In any one of paragraphs 11 to 13, The information related to the above behavior pattern includes at least one behavior element, and the at least one behavior element includes at least one of a first behavior element associated with a temporal element, a second behavior element associated with a spatial element, or a third behavior element associated with a situational element. A method wherein the number of the behavioral elements used to generate the second hint information is greater than the number of the behavioral elements used to generate the first hint information.

15. A non-transitory computer-readable storage medium storing instructions, wherein the instructions, when executed by at least a portion of at least one processor of an electronic device, cause the electronic device to: An action to generate correct answer information for information corresponding to a first user input; An operation of generating first hint information related to the correct answer information and information related to the user's behavior pattern using a first generative AI (artificial intelligence) model and providing the first hint information; An operation of obtaining a second user input including a first response (answer) to the first hint information; An operation of evaluating the first response based on the above correct answer information; and Based on the results of the above evaluation, the first generative AI model can be used to generate second hint information related to the correct answer information, information related to the behavior pattern, and first feedback information, and an action of providing the second hint information can be performed. A storage medium wherein the first feedback information includes at least one of the first response or the first hint information.

Images