Method for generating media content in extended reality environment, and electronic device therefor

Abstract

An electronic device according to one embodiment in the present document may comprise: one or more microphones; one or more cameras; at least one processor; and a memory for storing instructions. When executed by the at least one processor, the instructions can instruct the electronic device to: detect an event to be recorded while collecting activity-related data of a user; record, in response to the detection of the event, activity information acquired from the one or more microphones and / or the one or more cameras for a designated period of time; determine, on the basis of the activity-related data, whether a content generation condition is satisfied; determine a content generation item for producing media content if the content generation condition is satisfied; and combine at least some pieces of the recorded activity information on the basis of the determined content generation item so as to generate the media content. Other various embodiments identified through the present document are possible.

Description

Method for generating media content in an augmented reality environment and the electronic device thereof

[0001] The present disclosure relates to a method for generating media content in an augmented reality environment and an electronic device thereof.

[0002] As carrying electronic devices in daily life has become commonplace, these devices are being implemented in the form of multimedia equipment equipped with complex functions, such as taking photos or videos, playing music or video files, playing games, transmitting and receiving broadcasts, or supporting wireless communication connections. Recently, to further enhance the user experience and strengthen portability and convenience, electronic devices are evolving into new forms in terms of hardware and software.

[0003] Furthermore, with the advancement of artificial intelligence technology and the emergence of generative AI models, various attempts are being made to provide services for new content expanded from user input data, rather than being limited to the aforementioned input data.

[0004] The information described above may be provided as related art for the purpose of aiding understanding of the present disclosure. No claim or determination is made as to whether any of the foregoing may be applied as prior art related to the present disclosure.

[0005] An electronic device according to one embodiment of the present disclosure may include one or more microphones, one or more cameras, at least one processor, and a memory for storing instructions. The instructions may be executed by the at least one processor to enable the electronic device to detect a recording event while collecting user activity-related data. The instructions may be executed by the at least one processor to enable the electronic device to record activity information obtained from at least one of the one or more microphones or the one or more cameras for a specified time in response to the detection of the event. The instructions may be executed by the at least one processor to enable the electronic device to determine whether a content creation condition is satisfied based on the activity-related data. The instructions may be executed by the at least one processor to enable the electronic device to determine a content creation item for the production of media content if the content creation condition is satisfied. The above instructions may be executed by the at least one processor to enable the electronic device to generate media content by combining at least a portion of the recorded activity information based on the determined content generation item. The above instructions may be executed by the at least one processor to enable the electronic device to store the generated media content.

[0006] A method according to one embodiment of the present disclosure may include an operation of detecting a recording target event while collecting user activity-related data. The method may include an operation of recording activity information obtained using at least one of one or more microphones, one or more cameras, or one or more sensors for a specified period of time in response to the detection of the event. The method may include an operation of determining whether a content creation condition is satisfied based on the activity-related data. If the content creation condition is satisfied, the method may include an operation of determining a content creation item for producing media content. The method may include an operation of creating media content by combining at least a portion of the recorded activity information based on the determined content creation item. The method may include an operation of storing the created media content.

[0007] A computer-readable recording medium according to one embodiment disclosed in this document may include programs executable on a computer. The programs may perform an operation of detecting a recording target event while collecting user activity-related data. In response to the detection of the event, the programs may perform an operation of recording activity information acquired using at least one of one or more microphones, one or more cameras, or one or more sensors for a specified period of time. The programs may perform an operation of determining whether a content creation condition is satisfied based on the activity-related data. If the content creation condition is satisfied, the programs may perform an operation of determining a content creation item for producing media content. The programs may perform an operation of creating media content by combining at least a portion of the recorded activity information based on the determined content creation item. The programs may perform an operation of storing the created media content.

[0008] FIG. 1 is a block diagram of an electronic device in a network environment according to one embodiment.

[0009] FIG. 2 is a diagram illustrating the configuration of an electronic device according to one embodiment.

[0010] FIG. 3 is a drawing illustrating the detailed configuration of an electronic device according to one embodiment.

[0011] FIGS. 4, FIGS. 5, FIGS. 6, FIGS. 7 and FIGS. 8 are drawings illustrating a method of detecting a recording target event in an electronic device according to various embodiments.

[0012] FIG. 9 is a flowchart illustrating a method of operation of an electronic device according to one embodiment.

[0013] FIGS. 10, FIGS. 11, FIGS. 12, FIGS. 13 and FIGS. 14 are drawings illustrating a method of generating media content regarding user activity in an electronic device according to various embodiments.

[0014] FIG. 15 is a diagram illustrating a method of generating media content by an electronic device interacting with an external electronic device and / or an external server according to one embodiment.

[0015] FIG. 16 is a drawing showing a system including a generative artificial intelligence model according to one embodiment.

[0016] In relation to the description of the drawings, the same or similar reference numerals may be used for identical or similar components.

[0017] Hereinafter, various embodiments disclosed in this document are described with reference to the attached drawings. It should be understood that the various embodiments of this disclosure are not intended to limit the embodiments of this disclosure to a specific form and include various modifications, equivalents, and / or alternatives of this disclosure.

[0018] Electronic devices can collect data related to a user's behavior or psychological state and generate and provide records about the individual user based on the collected data. For example, electronic devices can use a journaling application that can collect and edit user activity data to recommend journaling items to be recorded and generate personalized journaling prompts. In this case, since items written directly by the user or selected by the user are primarily recorded, there is difficulty in recording items not selected by the user other than those presented in the prompt, and there may be limitations in generating only standardized types of content.

[0019] Various embodiments of this document may provide various embodiments for analyzing various data (e.g., biometric data, location data, video / audio data, device data, etc.) collected in response to user activity while wearing an electronic device to detect an event recognized as an activity different from the usual, and for generating and providing various types of media content based on at least a portion of the user activity information recorded in response to the detection of said event.

[0020] The technical tasks intended to be accomplished in this document are not limited to those mentioned above, and other technical tasks not mentioned will be clearly understood by those skilled in the art to which this document belongs.

[0021] FIG. 1 is a drawing illustrating an electronic device in a network environment (100) according to one embodiment.

[0022] Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) through a first network (198) (e.g., a short-range wireless communication network) or with at least one of an electronic device (104) or a server (108) through a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) through a server (108). According to one embodiment, the electronic device (101) may include a processor (120), memory (130), input module (150), sound output module (155), display module (160), audio module (170), sensor module (176), interface (177), connection terminal (178), haptic module (179), camera module (180), power management module (188), battery (189), communication module (190), subscriber identification module (196), or antenna module (197). In some embodiments, at least one of these components (e.g., connection terminal (178)) may be omitted from the electronic device (101), or one or more other components may be added. In some embodiments, some of these components (e.g., sensor module (176), camera module (180), or antenna module (197)) may be integrated into a single component (e.g., display module (160)).

[0023] The processor (120) can control at least one other component (e.g., hardware or software component) of the electronic device (101) connected to the processor (120) by executing software (e.g., program (140)), and can perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor (120) can store commands or data received from other components (e.g., sensor module (176) or communication module (190)) in volatile memory (132), process the commands or data stored in volatile memory (132), and store the resulting data in non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., central processing unit or application processor) or an auxiliary processor (123) that can operate independently or together with it (e.g., graphics processing unit, neural processing unit (NPU), image signal processor, sensor hub processor, or communication processor). For example, if the electronic device (101) includes a main processor (121) and an auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a designated function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as part thereof.

[0024] The auxiliary processor (123) may control at least some of the functions or states associated with at least one component of the electronic device (101) (e.g., display module (160), sensor module (176), or communication module (190)) on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. According to one embodiment, the auxiliary processor (123) (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module (180) or communication module (190)). According to one embodiment, the auxiliary processor (123) (e.g., neural network processing unit) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or through a separate server (e.g., server (108)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers.An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may include a software structure, either additionally or substantially.

[0025] The memory (130) can store various data used by at least one component of the electronic device (101) (e.g., processor (120) or sensor module (176)). The data may include, for example, input data or output data for software (e.g., program (140)) and related commands. The memory (130) may include volatile memory (132) or non-volatile memory (134).

[0026] The program (140) may be stored as software in memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0027] The input module (150) can receive commands or data to be used for a component of the electronic device (101) (e.g., processor (120)) from outside the electronic device (101) (e.g., user). The input module (150) may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0028] The sound output module (155) can output a sound signal to the outside of the electronic device (101). The sound output module (155) may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as multimedia playback or recording playback. The receiver may be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part thereof.

[0029] The display module (160) can visually provide information to an external (e.g., user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling said device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of the force generated by said touch.

[0030] The audio module (170) can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150) or output sound through the sound output module (155) or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphones) connected directly or wirelessly to the electronic device (101).

[0031] The sensor module (176) can detect the operating state of the electronic device (101) (e.g., power or temperature) or the external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, the sensor module (176) may include, for example, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0032] The interface (177) may support one or more specified protocols that can be used for the electronic device (101) to be connected directly or wirelessly to an external electronic device (e.g., electronic device (102)). According to one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0033] The connection terminal (178) may include a connector through which the electronic device (101) can be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0034] The haptic module (179) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that the user can perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module (179) may include, for example, a motor, a piezoelectric element, or an electric stimulation device.

[0035] The camera module (180) can capture still images and video. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0036] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least part of a power management integrated circuit (PMIC).

[0037] The battery (189) can supply power to at least one component of the electronic device (101). According to one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0038] The communication module (190) can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between an electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may include one or more communication processors that operate independently of the processor (120) (e.g., application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., cellular communication module, short-range wireless communication module, or GNSS (global navigation satellite system) communication module) or a wired communication module (194) (e.g., LAN (local area network) communication module, or power line communication module). The corresponding communication module among these communication modules can communicate with an external electronic device (104) through a first network (198) (e.g., a short-range communication network such as Bluetooth, Wi-Fi (wireless fidelity) direct or IrDA (infrared data association)) or a second network (199) (e.g., a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can identify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) using subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module (196).

[0039] The wireless communication module (192) can support 5G networks and next-generation communication technologies following 4G networks, for example, new radio access technology. NR access technology can support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and connection of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low-latency communications (URLLC)). The wireless communication module (192) can support a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate, for example. The wireless communication module (192) can support various technologies for securing performance in the high-frequency band, such as beamforming, massive MIMO (multiple-input and multiple-output), full-dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large-scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), external electronic device (e.g., electronic device (104)), or network system (e.g., second network (199)). According to one embodiment, the wireless communication module (192) can support a Peak data rate (e.g., 20 Gbps or more) for eMBB realization, loss coverage (e.g., 164 dB or less) for mMTC realization, or U-plane latency (e.g., downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less) for URLLC realization.

[0040] An antenna module (197) can transmit a signal or power to an external source (e.g., an external electronic device) or receive it from an external source. According to one embodiment, the antenna module (197) may include an antenna comprising a radiator made of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as a first network (198) or a second network (199), may be selected from the plurality of antennas, for example, by a communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, other components (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module (197).

[0041] According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the printed circuit board and capable of supporting a specified high frequency band (e.g., mmWave band), and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top surface or side surface) of the printed circuit board and capable of transmitting or receiving a signal of the specified high frequency band.

[0042] At least some of the above components can be connected to each other via a communication method between peripheral devices (e.g., bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)) and exchange signals (e.g., commands or data) with each other.

[0043] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) through a server (108) connected to a second network (199). Each of the external electronic devices (102, or 104) may be the same or different type of device as the electronic device (101). According to one embodiment, all or part of the operations performed on the electronic device (101) may be performed on one or more of the external electronic devices (102, 104, or 108). For example, if the electronic device (101) needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device (101) may request one or more external electronic devices to perform at least part of the function or service instead of performing the function or service itself or additionally. One or more external electronic devices that receive the above request may execute at least part of the requested function or service, or additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may provide the result as is or additionally processed as at least part of the response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device (101) may provide ultra-low latency services using, for example, distributed computing or mobile edge computing. In one embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server using machine learning and / or neural networks. According to one embodiment, the external electronic device (104) or the server (108) may be included within the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0044] FIG. 2 is a drawing illustrating the configuration of an electronic device (200) according to one embodiment.

[0045] Referring to FIG. 2, the electronic device (200) is a device that generates media content based on user activity information recorded while being worn by a user, and may include one or more microphones (210), one or more cameras (220), one or more sensors (230), a communication circuit (240), a display (250), at least one processor (260), and a memory (270). In FIG. 2, the electronic device (200) may correspond to the electronic device (101) shown in FIG. 1. Depending on various embodiments, the electronic device (200) may be a wearable type device that can be worn on a part of the user's body (e.g., a head-mounted display (HMD) device or AR glasses), or a portable multimedia device that the user can carry (e.g., a smartphone or a tablet PC).

[0046] In one embodiment, one or more microphones (210) (e.g., input module (150) of FIG. 1) can acquire voice data spoken by the user or around the user or sound data generated around the user.

[0047] In one embodiment, one or more cameras (220) (e.g., camera module (180) of FIG. 1) can acquire image data of the user or the user's surroundings.

[0048] In one embodiment, one or more sensors (230) (e.g., sensor module (176) of FIG. 1) can acquire data related to the activity of a user wearing the electronic device (200). For example, one or more sensors (230) can acquire the user's biometric data (e.g., body temperature, heart rate, blood pressure, or exercise status). In another example, one or more sensors (230) can acquire location data indicating the user's current location.

[0049] In one embodiment, the communication circuit (240) (e.g., the communication module (190) of FIG. 1) may support a communication connection with an external electronic device. For example, the communication circuit (240) may establish a call connection with an external electronic device. Depending on various embodiments, the communication circuit (240) may establish a communication connection with an electronic device located within a predetermined distance using short-range communication such as Bluetooth, Wi-Fi Direct, or BLE (Bluetooth Low Energy), or may establish a communication connection with an external electronic device located outside a predetermined distance using long-range communication such as cellular or Wi-Fi to exchange data.

[0050] In one embodiment, the display (250) (e.g., the display module (160) of FIG. 1) may display information intended to be provided to the user. For example, the display (250) may display media content generated based on the user's activity information.

[0051] In one embodiment, the display (250) may be composed of at least one of an LCD (liquid crystal display), a TFT-LCD (thin film transistor LCD), an OLED (organic light emitting diodes), an LED, an AMOLED (active matrix organic LED), a flexible display, and a 3-dimensional display. Additionally, some of these displays may be configured to be transparent or light-transmitting so that the outside can be seen through them. This may be configured in the form of a transparent display including a TOLED (transparent OLED).

[0052] According to various embodiments, at least one display (250) may be in a form where the display for the user's left eye and the display for the right eye are separated, or it may be an integrated display in which the left and right displays are not separated.

[0053] In one embodiment, the memory (270) (e.g., memory (130) of FIG. 1) may store instructions that cause the electronic device (200) to perform various operations by at least one processor (260) (e.g., processor (120) of FIG. 1) at execution. For example, at least one processor (260) may record user activity information and control the performance of operations to create and provide media content by utilizing at least a portion of the recorded activity information.

[0054] In one embodiment, at least one processor (260) may collect activity-related data of the user while the user is wearing the electronic device (200). The activity-related data may include at least one of voice data, video data, biometric data, or location data associated with the user's activity. For example, at least one processor (260) may acquire voice data of the user or the surroundings of the user using one or more microphones (210) and acquire video data of the surroundings of the user using one or more cameras (220). In another example, at least one processor (260) may detect the user's biometric data (e.g., body temperature, heart rate, blood pressure, or exercise status) using one or more sensors (230) and acquire location data regarding the user's current location. According to various embodiments, at least one processor (260) may acquire the activity-related data from at least one external electronic device connected to the electronic device (200). For example, if at least one processor (260) is wirelessly connected to a first external electronic device through a communication circuit (240), the activity-related data can be obtained from the first external electronic device (e.g., a smartphone or a wearable type device). The collected activity-related data may be stored in memory (270) at least temporarily, and among the activity-related data, data determined to be a record subject may be stored in a separate area within memory (270), and data not a record subject may be deleted after a specified period of time.

[0055] In one embodiment, at least one processor (260) can determine whether a recording target event has occurred while collecting the activity-related data. At least one processor (260) can check whether activity-related data exceeding a specified range based on the user's average activity data is detected while collecting the activity-related data, and if activity-related data exceeding the specified range is detected, determine that the recording target event has occurred. According to various embodiments, the verification of the activity-related data may be performed using a first artificial intelligence model trained to determine whether the recording target event has occurred based on the collected activity-related data. At least one processor (260) can provide the collected activity-related data to the first artificial intelligence model to obtain a determination result regarding whether the event has occurred. If the event has occurred according to the determination result, at least one processor (260) can store the activity-related data obtained for a specified time from the time the event occurred as the user's activity information in a first designated area of ​​the memory (270) or in a separate storage space for recording activity information located outside the electronic device (200).

[0056] In one embodiment, at least one processor (260) may manage one or more activity information stored in a first designated area of ​​memory (270). At least one processor (260) may ensure that the one or more activity information is maintained or deleted by considering the storage capacity of the first designated area and / or a retention period set for each activity information. According to various embodiments, the management of the one or more activity information may be performed using a second artificial intelligence model trained to determine whether to delete the one or more activity information stored in the first designated area. At least one processor (260) may provide record-related data of the first designated area to the second artificial intelligence model to obtain a determination result regarding whether to delete each of the one or more activity information. Each activity information may be maintained or deleted in the first designated area according to the determination result.

[0057] In one embodiment, at least one processor (260) can determine whether a content creation condition is satisfied based on the collected activity-related data and a user command or a schedule command. The user command may correspond to a request to create media content entered by the user. The schedule command may correspond to a time or condition for creating media content specified by the user. For example, in response to identifying a user command requesting to record the current situation (e.g., touch of a shooting button), at least one processor (260) may record the activity-related data acquired from the time the user command was entered as the user's activity information and determine that the content creation condition for the recorded activity information is satisfied. In another example, if at least one processor (260) identifies a schedule command such as "write a picture diary every evening at 9 o'clock," it may determine that the content creation condition is satisfied at 9 o'clock and check the activity information for the corresponding day recorded up to 9 o'clock. In another example, if at least one processor (260) identifies a schedule command such as ‘make a short form video whenever A is met,’ it may determine that the content creation condition is satisfied if it detects A’s face from video data obtained from one or more cameras (220) or detects A’s voice from voice data obtained from one or more microphones (210).

[0058] In one embodiment, at least one processor (260) may determine a content creation item for producing media content when the content creation condition is satisfied. The content creation item may include at least one of the type of media content, file format, time of creation, or activity information used for creating the media content. According to various embodiments, the determination of the content creation condition and the determination of the content creation item may be performed using a third artificial intelligence model trained to determine the content creation item required to create / store the media content. At least one processor (260) may obtain the content creation item by providing the activity-related data, the user command, or the schedule command to the third artificial intelligence model.

[0059] In one embodiment, at least one processor (260) may generate the media content by combining at least a portion of the recorded activity information based on the content generation item. According to various embodiments, the generation of the media content may be performed using a fourth artificial intelligence model trained to generate media content regarding the user's activity. At least one processor (260) may obtain the media content by providing the content generation item obtained from the third artificial intelligence model to the fourth artificial intelligence model.

[0060] In one embodiment, at least one processor (260) can determine whether at least one person whose recording is not permitted is identified in the media content. For example, if at least one processor (260) identifies that the face of a person who has not consented to the disclosure of personal information is included in the media content, it may remove the face of the identified person from the media content or process the media content so that the face of the identified person is not visible. In another example, if at least one processor (260) identifies that a place whose exposure is not permitted is included in the media content, it may remove the portion containing the identified place from the media content or process the media content so that the identified place is not visible. In yet another example, at least one processor (260) may control so that data related to the user's activity is not recorded while the user is located in a designated place (e.g., a theater, a restroom, a conference room) or while the electronic device (200) is set to a designated mode (e.g., Do Not Disturb mode).

[0061] In one embodiment, at least one processor (260) may store the generated media content in a second designated area of ​​memory (270) or in a separate storage space for storing media content located outside the electronic device (200). The media content may be stored in a file format that can be viewed, reused, or shared on one or more external electronic devices. The media content stored in the second designated area may be managed separately from the activity information of the first designated area.

[0062] In one embodiment, when a request to view the media content is input by a user, at least one processor (260) may load the media content from the second designated area of ​​the memory (270) and provide it through the display (250). According to various embodiments, while providing the media content, at least one processor (260) may provide data related to activity information included in the media content. For example, at least one processor (260) may display summary information, such as the time and place at which the activity information is recorded, the user's biometric data, or an event that occurred, on the display (250) together with the media content.

[0063] FIG. 3 is a drawing illustrating the detailed configuration of an electronic device (200) according to one embodiment. Functions or operations described with reference to FIG. 3 may be understood as functions performed by at least one processor (260) of the electronic device (200) of FIG. 2. For example, the activity data receiving unit (315) and the content creation command checking unit (330) shown in FIG. 3 may be implemented as software modules containing at least one command. At least one processor (260) may execute commands (e.g., instructions) stored in memory (e.g., memory (130) of FIG. 1 or memory (270) of FIG. 2) to implement the software modules shown in FIG. 3, and may control hardware related to the function (e.g., microphone (210), camera (220), sensor (230), communication circuit (240), display (250) or memory (270) of FIG. 2). According to various embodiments, the electronic device (200) is not limited to the components shown in FIG. 3 and may additionally include components among the components shown in FIG. 1 that correspond to the functions required by the electronic device (200).

[0064] Referring to FIG. 3, the electronic device (200) may include an activity data receiving unit (315), an event judgment model (320), a data recording unit (325), a content creation command confirmation unit (330), a content judgment model (335), a content creation model (340), or a media content storage unit (345).

[0065] In one embodiment, the activity data receiving unit (315) may receive activity data (310) collected while being worn by a user from one or more microphones (input module (150) of FIG. 1 or microphone (210) of FIG. 2), one or more cameras (e.g., camera module (180) of FIG. 1 or camera (220) of FIG. 2) or one or more sensors (e.g., sensor module (176) of FIG. 1 or sensor (230) of FIG. 2). The activity data (310) is data detected in association with the user's behavior or psychological state and may include voice data obtained from the one or more microphones (210), image data obtained from the one or more cameras (220), biosignal data or location data obtained from the one or more sensors (230). According to various embodiments, the activity data receiving unit (315) may receive the activity data (310) from an external electronic device (e.g., the electronic device (102 or 104) of FIG. 1) that is communicationally connected to the electronic device (200).

[0066] In one embodiment, the event judgment model (320) can determine whether an event worth recording has occurred in relation to the user based on the received activity data (310). The event judgment model (320) is an artificial intelligence model trained to determine whether to record the activity data (310). If activity data exceeding a specified range based on the user's average activity data is detected, the model determines that an event to be recorded, which is different from the user's usual activity, has occurred and decides to record the activity data (310) received from that point in time. According to various embodiments, the event judgment model (320) is pre-trained by an external server based on a data set regarding the average activity of ordinary people, and in the initial stages of use, it can determine whether the event to be recorded has occurred based on the pre-trained general data set. Subsequently, when the electronic device (200) is used for more than a specified time and activity data personalized to the user is accumulated, the event judgment model (320) can determine whether the event to be recorded has occurred based on the accumulated personalized activity data.

[0067] In one embodiment, when the data recording unit (325) obtains a determination result from the event determination model (320) that a recording target event has occurred, it may store the activity data (310) received for a specified period from the time of occurrence of the event as the user's activity information. The data recording unit (325) may store the user's activity information in a first designated area within the memory of the electronic device (200) (e.g., memory (130) of FIG. 1 or memory (270) of FIG. 2), or store it in a separate storage space for recording activity information located outside the electronic device (200).

[0068] According to various embodiments, the activity data receiving unit (315) may store activity data (310) based on a user command (301) requesting a record of the current situation. In this case, when the activity data receiving unit (315) confirms the user command (301), it may provide the activity data (310) received from the time of confirmation of the user command (301) to the event judgment model (320). The event judgment model (320) may determine the current situation as an event to be recorded, and the data recording unit (325) may store the activity data (310) received from the time of confirmation of the user command (301) as the user's activity information in a first designated area of ​​the memory (270) or in a separate storage space outside the electronic device (200), according to the judgment result obtained from the event judgment model (320).

[0069] In one embodiment, the content creation command confirmation unit (330) can confirm a user command (301) requesting the creation of media content input by the user or a schedule command (303) specifying the conditions for the creation of the media content. For example, the user command (301) may correspond to a selection on a menu / button requesting the recording of the user's activity data or requesting the creation of media content using the recorded activity information. As another example, the schedule command (303) may specify the conditions or timing for creating the media content.

[0070] In one embodiment, the content judgment model (335) can determine whether the content creation condition is satisfied based on the received activity data (310), the user command (301), or the schedule command (303), and if the content creation condition is satisfied, determine the content creation item for media content creation. The content judgment model (335) is an artificial intelligence model trained to determine the content creation items required to create media content, and can identify the content creation condition specified by the user (e.g., content type, creation time, or creation cycle) by analyzing the user command (301) or the schedule command (303) based on natural language processing. The content judgment model (335) can determine the type of media content to be created, the file format, the creation time, or the target information used for the creation of the media content among the user's activity information stored by the data recorder (325).

[0071] In one embodiment, the content generation model (340) can generate media content based on content generation items obtained from the content judgment model (335). The content generation model (340) is an artificial intelligence model trained to generate media content regarding user activities, and can generate the media content by combining / editing target information determined according to the content generation items among the user activity information stored by the data recorder (325).

[0072] In one embodiment, the media content storage unit (345) can store the media content obtained from the content creation model (340). The media content storage unit (345) can store the media content in a second designated area within the memory (270) of the electronic device (200), or store it in a separate storage space for storing media content located outside the electronic device (200).

[0073] In one embodiment, user activity information recorded by the data recording unit (325) and media content recorded by the media content storage unit (345) can be provided to the user through the display (250).

[0074] FIGS. 4, FIGS. 5, FIGS. 6, FIGS. 7 and FIGS. 8 are drawings illustrating a method of detecting a recording target event in an electronic device (200) according to various embodiments. The electronic device (200) may acquire activity data associated with the user's activity state while being worn by the user. The activity data may include at least one of voice data (e.g., voice spoken by the user or sound generated around the user) acquired from one or more microphones (e.g., input module (150) of FIG. 1 or microphone (210) of FIG. 2), image data (e.g., an image corresponding to the user's field of view) acquired from one or more cameras (e.g., camera module (180) of FIG. 1 or camera (220) of FIG. 2), user's biosignal data (e.g., body temperature, heart rate, blood pressure, or exercise state) or location data (e.g., the user's current location) acquired from one or more sensors (e.g., sensor module (176) of FIG. 1 or sensor (230) of FIG. 2). When the electronic device (200) is connected to at least one external device (e.g., a wearable device), the electronic device (200) may acquire the activity data from the at least one external device. While acquiring the activity data, the electronic device (200) determines whether an event different from the user's usual activity has occurred, and if it is determined that the event has occurred, it may determine the activity data acquired from the time the event occurred as a recording target and record / store it as the user's activity information. According to various embodiments, the electronic device (200) may determine whether the event has occurred based on the user's emotions predicted based on the activity data. For example, the electronic device (200) may predict that the event has occurred based on the content of the conversation or the location with the person (e.g., family, friends, colleagues) the user met, identified based on the user's activity data.As another example, the electronic device (200) may determine that the event has occurred when activity data associated with a person or place previously recorded as positive emotion is detected, and may record / store the user's activity information.

[0075] Referring to FIG. 4, while the user (400) is wearing the wearable device (410), the wearable device (410) can acquire activity data of the user (400). The wearable device (410) may correspond to the electronic device (101) of FIG. 1 or the electronic device (200) of FIG. 2. Depending on various embodiments, the wearable device (410) may be implemented as a head-mounted display (HMD) device that can be worn on the user's head or as AR glasses in the form of glasses. For example, the wearable device (410) can confirm that the user (400) performs a gesture (403) corresponding to an OK sign using their hand based on image data acquired from one or more cameras (220) and / or biosignal data (e.g., motion recognition data) acquired from a smartwatch (420) worn on the user's wrist. In another example, the wearable device (410) can identify the sound (405) of a user (400) laughing happily based on voice data obtained from one or more microphones (210).

[0076] In one embodiment, the wearable device (410) may determine a current enjoyable moment as a recording target event based on a gesture (403) and / or voice (405) recognized by the user (400). The wearable device (410) may record video data (450) from the time when the user (400)'s current situation is determined as the recording target event as activity information of the user (400). The video data (450) may be stored in a designated area within a memory (e.g., memory (130) of FIG. 1 or memory (270) of FIG. 2) or in a separate storage space for recording activity information outside the wearable device (410), and may be utilized when creating media content regarding the user's (400) activities thereafter.

[0077] In one embodiment, the wearable device (410) can determine whether the image data (450) includes a person who is not allowed to be recorded. If it is determined that the image data (450) includes a person who has not consented to the disclosure of personal information or is unrelated to the recorded situation, the wearable device (410) can remove or make invisible the face of the person identified in the image data (450) and store it.

[0078] Referring to FIG. 5, while a user (500) is watching a soccer match while wearing a wearable device (510), the wearable device (510) can acquire activity data of the user (500). The wearable device (510) may correspond to the electronic device (101) of FIG. 1 or the electronic device (200) of FIG. 2. For example, the wearable device (510) can confirm that the user (500) performs a cheering gesture (503) based on biosignal data (e.g., motion recognition data) acquired from a smart watch (520) worn on the user's wrist during a goal-scoring scene while the user (500) is watching a soccer match. In another example, the wearable device (510) can identify the sound (505) of the user (500) shouting and cheering loudly based on voice data obtained from one or more microphones (210), and detect that the user's (500) body temperature and heart rate are rising rapidly based on biosignal data (e.g., body temperature or heart rate) obtained from one or more sensors (230).

[0079] In one embodiment, the wearable device (510) can determine the scene currently being viewed as a recording target event based on the gesture (503), voice (505), and biosignal recognized by the user (500). At the time the wearable device (510) determines the current scene as the recording target event, it can record the video data (550) currently being viewed and the voice data of the user (500) obtained from one or more microphones as activity information of the user (500). The video data (550) and the voice data may be stored in a designated area within the memory (270) or in a separate storage space for recording activity information outside the wearable device (510), and may be utilized when creating media content regarding the user's (500) activities thereafter.

[0080] In one embodiment, the wearable device (510) determines that the image data (550) is video content that is permitted to be disclosed, and recognizes that the people included in the image data (550) are not subject to personal information protection, and can store the image data (550) without any special processing.

[0081] Referring to FIG. 6, while a user (600) is walking down the street while wearing a wearable device (610), the wearable device (610) can acquire activity data of the user (600). The wearable device (610) may correspond to the electronic device (101) of FIG. 1 or the electronic device (200) of FIG. 2. For example, the wearable device (610) can detect that a situation (605) has occurred in which the user (600) has a bag held in their hand stolen by an unidentified man, based on image data acquired from one or more cameras (220). As another example, the wearable device (610) can detect that the user's (600) heart rate has risen rapidly based on biosignal data (e.g., heart rate) acquired from one or more sensors (230).

[0082] In one embodiment, the wearable device (610) may determine a current theft incident as a recording event based on a situation (605) and / or biosignals recognized for the user (600). From the time the wearable device (610) detects the user's (600) current situation as a recording event, it may record image data (650) obtained from one or more cameras (220) and biosignals and location data obtained from one or more sensors (230) as activity information of the user (600). The image data (650), the biosignals, and the location data may be stored in a designated area within a memory (e.g., memory (130) in FIG. 1 or memory (270) in FIG. 2) or in a separate storage space for recording activity information outside the wearable device (610), and may be utilized when creating media content regarding the user's (600) activities.

[0083] Referring to FIG. 7, while a user (700) is watching a video while wearing a wearable device (710), the wearable device (710) can detect a recording target event that the user (700) has pre-set. The wearable device (710) may correspond to the electronic device (101) of FIG. 1 or the electronic device (200) of FIG. 2. For example, if the user (700) sets a recording target event for a child's unusual or dangerous behavior, the wearable device (710) can confirm that a dangerous situation (705) in which the child falls into a basket has occurred based on video data obtained from one or more cameras (220).

[0084] In one embodiment, the wearable device (710) may determine that the identified dangerous situation (705) corresponds to a recording target event set by the user. From the time the wearable device (710) detects the dangerous situation (705) as the recording target event, it may record image data (750) obtained from one or more cameras (220) as activity information of the user (700). The image data (750) may be stored in a designated area within a memory (e.g., memory (130) of FIG. 1 or memory (270) of FIG. 2) or in a separate storage space for recording activity information outside the wearable device (710), and may be utilized when creating media content regarding the user's (700) activities.

[0085] Referring to FIG. 8, while a user (800) is traveling on a train while wearing a wearable device (810), the wearable device (810) can detect user input requesting the user (800) to record the current situation. The wearable device (810) may correspond to the electronic device (101) of FIG. 1 or the electronic device (200) of FIG. 2. For example, when the wearable device (810) detects user input (805) in which the user (800) presses a recording button (815) on the screen provided through the display of the wearable device (810), the wearable device (810) can determine the surrounding environment and current situation in which the user (800) is traveling as events to be recorded based on video data acquired from one or more cameras (220).

[0086] In one embodiment, the wearable device (810) may record video data (850) obtained from one or more cameras (220) and voice data obtained from one or more microphones (210) as activity information of the user (800) from the time the user input (805) is detected. The video data (850) and the voice data may be stored in a designated area within a memory (e.g., memory (130) of FIG. 1 or memory (270) of FIG. 2) or in a separate storage space for recording activity information outside the wearable device (810), and may be utilized when creating media content regarding the user's (800) activities.

[0087] FIG. 9 is a flowchart illustrating a method of operation of an electronic device (200) according to one embodiment. According to one embodiment, the electronic device (200) is a device that generates media content based on user activity information recorded while being worn by a user, and may correspond to the electronic device (101) shown in FIG. 1. The operations of FIG. 9 may be performed by a processor included in the electronic device (200) (e.g., the processor (120) of FIG. 1 or the processor (260) of FIG. 2).

[0088] Referring to FIG. 9, in operation 910, the electronic device (200) can detect a recording target event based on user activity-related data collected while the user is wearing the electronic device (200). The activity-related data may include at least one of voice data, video data, biometric data, or location data associated with the user's activity. For example, the electronic device (200) may acquire voice data of the user or the surroundings of the user using one or more microphones (e.g., the input module (150) of FIG. 1 or the microphone (210) of FIG. 2), and acquire video data of the surroundings of the user using one or more cameras (e.g., the camera module (180) of FIG. 1 or the camera (220) of FIG. 2). As another example, the electronic device (200) may detect the user's biometric data (e.g., body temperature, heart rate, blood pressure, or exercise state) using one or more sensors (e.g., the sensor module (176) of FIG. 1 or the sensor (230) of FIG. 2), and acquire location data regarding the user's current location. In another example, if the electronic device (200) is wirelessly connected to a first external electronic device through a communication circuit (e.g., the communication module (190) of FIG. 1 or the communication circuit (240) of FIG. 2), the activity-related data can be obtained from the first external electronic device (e.g., a smartphone or a wearable device). The collected activity-related data may be stored in memory (270) at least temporarily, and among the activity-related data, data determined to be a recording target may be stored in a separate area within memory (270), and data not a recording target may be deleted after a specified period of time.

[0089] According to one embodiment, the electronic device (200) checks whether activity-related data exceeding a specified range based on the user's average activity data is detected during operation 910 while collecting the activity-related data, and if activity-related data exceeding the specified range is detected, it can determine that the event to be recorded has occurred. According to various embodiments, operation 910 may be performed using a first artificial intelligence model trained to determine whether the event to be recorded has occurred based on the collected activity-related data. The electronic device (200) may provide the collected activity-related data to the first artificial intelligence model to obtain a determination result regarding whether the event has occurred.

[0090] According to one embodiment, in operation 920, if the electronic device (200) determines that the event has occurred based on the judgment result obtained from the first artificial intelligence model, it may record activity-related data obtained for a specified time from the time the event occurred as the user's activity information. In operation 920, the electronic device (200) may store the activity information in a first designated area of ​​memory (e.g., memory (130) of FIG. 1 or memory (270) of FIG. 2) or in a separate storage space for recording activity information outside the electronic device (200).

[0091] According to one embodiment, in operation 930, the electronic device (200) can determine whether the conditions for creating content are satisfied based on the collected activity-related data and user commands or schedule commands. The user command may correspond to a request to create media content entered by the user. The schedule command may correspond to a time or condition for creating media content specified by the user. For example, in response to identifying a user command requesting to record the current situation (e.g., touch of a shooting button), the electronic device (200) may record the activity-related data acquired from the time the user command was entered as the user's activity information and determine that the conditions for creating content for the recorded activity information are satisfied. In another example, if the electronic device (200) identifies a schedule command such as "write a picture diary every evening at 9 o'clock," it may determine that the conditions for creating content are satisfied at 9 o'clock and check the activity information for the corresponding day recorded up to 9 o'clock. In another example, when the electronic device (200) identifies a schedule command such as ‘make a short form video whenever A is met,’ it may determine that the content creation condition is satisfied if it detects A’s face from video data obtained from one or more cameras (220) or detects A’s voice from voice data obtained from one or more microphones (210).

[0092] According to one embodiment, in operation 940, the electronic device (200) may determine a content creation item for producing media content when the content creation condition is satisfied. The content creation item may include at least one of the type of media content, file format, time of creation, or activity information used for creating the media content. According to various embodiments, operations 930 and 940 may be performed using a third artificial intelligence model trained to determine a content creation item required to create / store the media content. The electronic device (200) may obtain the content creation item by providing the activity-related data, the user command, or the schedule command to the third artificial intelligence model.

[0093] According to one embodiment, in operation 950, the electronic device (200) may generate the media content by combining at least a portion of the recorded activity information based on the content generation item. According to various embodiments, operation 950 may be performed using a fourth artificial intelligence model trained to generate media content regarding the user's activity. The electronic device (200) may obtain the media content by providing the content generation item obtained from the third artificial intelligence model to the fourth artificial intelligence model. In operation 950, the electronic device (200) may store the generated media content in a second designated area of ​​memory (270) or in a separate storage space for storing media content outside the electronic device (200). The media content may be stored in a file format that can be viewed, reused, or shared on one or more external electronic devices. The media content stored in the second designated area may be managed separately from the activity information in the first designated area.

[0094] According to various embodiments, the electronic device (200) may determine whether at least one person whose recording is not permitted is identified in the media content. For example, if the electronic device (200) identifies that the face of a person who has not consented to the disclosure of personal information is included in the media content, it may remove the face of the identified person from the media content or process the media content so that the face of the identified person is not visible. In another example, if the electronic device (200) identifies that a place where exposure is not permitted is included in the media content, it may remove the portion containing the identified place from the media content or process the media content so that the identified place is not visible.

[0095] According to one embodiment, after operation 950, when a user inputs a request to view the media content, the electronic device (200) can load the media content from the second designated area of ​​the memory (270) or a separate storage space for storing media content outside the electronic device (200) and provide it through a display (e.g., the display module (160) of FIG. 1 or the display (250) of FIG. 2).

[0096] FIGS. 10, 11, 12, 13, and 14 illustrate a method of generating media content regarding user activity in an electronic device (200) according to various embodiments. The electronic device (200) can determine whether conditions for generating content are met based on activity-related data collected while being worn by a user and user commands or schedule commands. The user commands request the creation of media content input by the user, and the schedule commands may relate to the time or conditions for generating media content specified by the user. The electronic device (200) can analyze the collected activity-related data to identify the user's current situation and determine whether it corresponds to the conditions for generating content (or the time of creation) specified by the identified user commands or schedule commands. If the electronic device (200) determines that the conditions for generating content are met, it can generate media content by determining the content creation items necessary for producing media content. The above content creation item may include at least one of the target information used for creating the media content, such as the type of media content to be created, the file format, the time of creation, or activity information recorded for the user.

[0097] Referring to FIG. 10, the electronic device (200) can determine the conditions for creating content (or the time of creation) desired by the user based on a user command or a schedule command entered by the user. For example, when the electronic device (200) receives a user command such as "Write today's diary," it determines the time of receiving the user command as the time of content creation and can create media content (1010) of the descriptive diary type using at least some of the activity information recorded on that date. As another example, when the electronic device (200) checks a schedule command such as "Write a picture diary every day at 10 PM," it determines 10 PM designated by the schedule command as the time of content creation and can create media content (1020) of the picture diary type based on at least some of the activity information recorded up to 10 PM on that date.

[0098] In one embodiment, the electronic device (200) may identify voice data recorded on a corresponding date based on the determined content creation time and generate a narrative diary-type media content (1010) by summarizing the content of a lecture or conversation with a colleague included in the voice data. According to various embodiments, the electronic device (200) may analyze the user's mood or psychological state based on the user's biometric data recorded in correspondence with the time of acquisition of the voice data and reflect a color or font representing the analyzed mood / psychological state in the narrative diary-type media content (1010).

[0099] In one embodiment, the electronic device (200) identifies video data and / or voice data recorded on a corresponding date based on the determined content creation time, and can generate a picture diary type media content (1020) based on the content of a conversation with a friend included in the video data and / or voice data. The electronic device (200) analyzes the user's mood or psychological state based on the user's biometric data recorded in correspondence with the acquisition time of the video data and / or voice data, and can reflect a character reflecting the analyzed mood / psychological state in the picture diary type media content (1020).

[0100] Referring to FIG. 11, the electronic device (200) can determine the conditions for creating content (or the time of creation) desired by the user based on user commands or schedule commands entered by the user. For example, when the electronic device (200) confirms a schedule command such as "create content regarding activities for the last month at the end of each month," it determines the time specified on the last day of each month designated by the schedule command as the time of content creation and can create monthly diary-type media content (1110) based on at least some of the activity information recorded during the last month up to that date. As another example, when the electronic device (200) receives a user command such as "make a family trip album for last month," it determines the time at which the user command was received as the time of content creation and can create family album-type media content (1120) based on at least some of the activity information recorded during the family trip period.

[0101] In one embodiment, the electronic device (200) can identify a major activity record based on at least one of video data, voice data, biometric data, or location data recorded during the last month based on the determined content creation time, and generate the monthly diary type media content (1110) by combining a picture, emoticon, or simple sentence associated with the identified major activity record and creating it in a calendar format.

[0102] In one embodiment, the electronic device (200) may identify video data and / or location data recorded during a family trip period specified by the user command, and may combine / edit at least a portion of the identified video data and / or location data to generate the family album type media content (1120). Depending on various embodiments, the electronic device (200) may also incorporate colors or emoticons that match the style of the album or the family atmosphere into the family album type media content (1120).

[0103] Referring to FIG. 12, the electronic device (200) can generate 3D media content (1210, 1220) based on at least a portion of activity information recorded for the user. For example, the electronic device (200) can identify video data of enjoyable time spent with a child in the recorded activity information and generate 3D interactive type media content (1210) using the identified video data. As another example, the electronic device (200) can identify video data of a 3D experiential game in the recorded activity information and generate 3D short-form type media content (1220) by editing highlight portions of the identified video data.

[0104] Referring to FIG. 13, when the electronic device (200) receives a user command such as "Show me travel photos with X," it may identify video data corresponding to the travel period with X from activity information recorded for the user, and edit the identified video data to create media content (1310) of the image album type or video type. The electronic device (200) may analyze the user's mood or atmosphere at that time based on voice data or biometric data corresponding to the travel period, and edit the color representing the analyzed mood / atmosphere to reflect it in the media content (1310) of the image album type (or video type). According to various embodiments, the electronic device (200) may additionally identify other video data containing X from the recorded activity information, and create and provide album-type media content (1320) by combining / editing the additionally identified video data as associated content of the media content (1310). The electronic device (200) may create and provide the media content (1310) using video data containing a user and X obtained from the electronic device of X, or may create / provide the video data as associated content of the media content (1310).

[0105] Referring to FIG. 14, when the electronic device (200) checks a schedule command regarding an 'experiment-related meeting,' it can identify video data, audio data, or previously shared experiment-related data related to the experiment from activity information recorded for the user, and generate media content (1400) of the presentation material type customized for the target of sharing based on the identified data. For example, the electronic device (200) can generate media content (1400) of the presentation material type that is easy to share by combining / editing previously shared experiment content (1410), content regarding the progress of the experiment (1420), content regarding experiment conditions (1430), and / or content regarding a summary of experiment results (1440) related to the experiment.

[0106] FIG. 15 is a diagram illustrating a method of generating media content by an electronic device (200) interacting with an external electronic device (1510) and / or an external server (1520) according to one embodiment. According to various embodiments, the electronic device (200) may interact with a communication-connected external electronic device (1510) and / or an external server (1520) to prevent an overload of the electronic device (200)'s processing work during the process of generating media content regarding user activity.

[0107] Referring to FIG. 15, at least some of the functions / operations for generating the media content in the electronic device (200) may be performed by an external electronic device (1510) and / or a server (1520). The external electronic device (1510) may be a smartphone or wearable device that is connected via communication to the electronic device (200) through a communication circuit (e.g., the communication module (190) of FIG. 1 or the communication circuit (240) of FIG. 2) and has the same user account set as the electronic device (200). The server (1520) may be a server that supports the creation / management of the media content.

[0108] In one embodiment, an external electronic device (1510) may perform activity data collection (1511), activity information recording (1512), communication function support (1513), or media content creation (1514). The external electronic device (1510) may collect (1511) data associated with the user's activity (e.g., voice data, video data, biometric data, or location data). An electronic device (200) may obtain the collected activity data from the external electronic device (1510) through a communication circuit (240). The external electronic device (1510) may record (1512) data among the data associated with the user's activity that the electronic device (200) determines needs to be recorded as the user's activity information. The electronic device (200) may request at least a portion of the recorded activity information from the external electronic device (1510) when creating media content, and may obtain at least a portion of the recorded activity information from the external electronic device (1510) in response to the request. The external electronic device (1510) may support (1513) a communication connection between the electronic device (200) and the server (1520). If a direct connection with the server (1520) is not feasible, the electronic device (200) may obtain data / information necessary for creating media content from the server (1520) through the external electronic device (1510). The external electronic device (1510) may create media content (1514) using at least a portion of the recorded user activity information or the activity information recorded by the electronic device (200). The electronic device (200) may acquire the generated media content from the external electronic device (1510) and store it in a designated area within the memory (e.g., the memory (130) of FIG. 1 or the memory (270) of FIG. 2), or receive and provide the media content stored in the external electronic device (1510) when the media content is requested by the user.

[0109] In one embodiment, the server (1520) may perform activity information storage (1521), policy and command storage / management (1522), learning data storage and model training (1523), or media content generation (1524). The server (1520) may store (1521) user activity information recorded in the electronic device (200) and / or external electronic device (1510). The electronic device (200) may transfer at least a portion of the activity information recorded by the electronic device (200) and / or external electronic device (1510) to the server (1520) for storage / management, taking into account the storage capacity of the area for storing the activity information. The server (1520) may store and manage (1522) policies (e.g., policies regarding privacy protection or content sharing / management) and / or commands specified by the user that may be applied during the process of generating media content regarding user activity. The electronic device (200) can check the policy and / or command from the server (1520) when generating the media content. The server (1520) can store training data for training an artificial intelligence model used to generate the media content and can train the artificial intelligence model (1523) based on the training data. The electronic device (200) can obtain from the server (1520) a first artificial intelligence model trained to determine whether a recording target event occurs (e.g., the event judgment model (320) of FIG. 3), a second artificial intelligence model trained to determine whether one or more activity information is deleted, a third artificial intelligence model trained to determine content creation items required to generate / save media content (e.g., the content judgment model (335) of FIG. 3), or a fourth artificial intelligence model trained to generate media content regarding user activities (e.g., the content creation model (340) of FIG. 3). The server (1520) can generate media content (1524) using at least a portion of the activity information recorded by the electronic device (200) and / or the external electronic device (1510).The electronic device (200) may acquire the generated media content from the server (1520) and store it in a designated area within the memory (270), or receive and provide the media content stored in the server (1520) when the media content is requested by a user.

[0110] FIG. 16 is a drawing showing a system including a generative artificial intelligence model according to one embodiment.

[0111] Referring to FIG. 16, the User Query / Response Interface (1610) can receive user input. The user input may be in the form of natural language, images, and / or videos. Additionally, context information may be transmitted along with the user input. Context information may include various additional information at the time of user input. For example, information about the application currently being used by the user or the user's location information. Furthermore, user input may be in a mixed form of the aforementioned natural language, images, sounds, and context information. Additionally, user input may be in a non-natural language form, such as selecting a menu. The User Query / Response Interface (1610) can output results from a generative artificial intelligence system to the user. The output may be in the form of natural language or specific content, and may also be provided in the form of actions requested by the user.

[0112] The AI ​​framework (1620) receives user input and can coordinate and control each component necessary to perform the user's intent based on the user's query.

[0113] User input received from the User Query / Response Interface (1610) can be sent to the Prompt design component (1621). The Prompt design component (1621) can be used to generate a prompt suitable for inputting user input into a Large Language Model (LLM) or Large Multimodal Models (LMM). The Prompt design component (1621) may be an AI component that uses machine learning algorithms or neural networks to develop better prompts over time. Based on user input, the Prompt design component (1621) can generate a prompt by accessing a knowledge component (e.g., knowledge repositories (1640)) containing user preference data, a prompt library, and prompt examples, and can pass the generated prompt to the LLM or LMM.

[0114] The API / Plug-in management component (1623) can perform the role of communicating with external information when there is a request for additional information when passing user input as input to a generative model. The API / Plug-in management component (1623) establishes a channel to communicate with the outside of the AI ​​Interface via the API, and through the established channel, it can enable access to various data sources (e.g., knowledge repositories (1640)). Additionally, the API / Plug-in management component (1623) can request the application / service component (1630) via the API to perform an action that executes the user input as a final step, rather than an intermediate result, when the application or service needs to perform such action. The information obtained from the outside can be used to generate a prompt in the Prompt design component (1621) along with the user input, or it can be passed as input to the generative model.

[0115] The Refiner component (e.g., output modification component (1625)) can fine-tune the output of a generative model. For example, the Refiner component can verify whether the content generated by LLM and / or LMM is irrelevant, contains biased content, or includes harmful content. Additionally, the Refiner component can determine the extent to which the output matches the user's desired outcome and, if necessary, proceed with additional processing. Furthermore, the Refiner component can configure and provide hints to the user to help avoid unwanted outputs.

[0116] A Generative AI Model (1650) generally refers to an artificial intelligence neural network that generates new forms of data based on user input information. A Generative AI Model (1650) may include a model that generates images and / or a model that generates language. Models that generate images include, but are not limited to, GANs (generative adversarial networks) and VAEs (variational autoencoders), and examples include Diffusion-based generative models that use VAEs and Transformer structures. Models that generate language are models trained to output the most statistically appropriate output value based on input values, and examples include models such as CHAT-GPT 3 and CHAT-GPT 4. There are also LMMs that can recognize various forms of data input, such as text, images, and voice, and generate new data corresponding to them.

[0117] According to one embodiment, the electronic device (101, 200) of FIGS. 1 to 15 may be configured to include at least some of the User Query / Response Interface (1610) AI framework (1620), application / service component (1630), knowledge repositories (1640), or Generative AI Model (1650) of FIG. 16. According to one embodiment, at least some of the User Query / Response Interface (1610) AI framework (1620), application / service component (1630), knowledge repositories (1640), or Generative AI Model (1650) of FIG. 16 may be included in another electronic device (e.g., a server).

[0118] In one embodiment, an electronic device (e.g., electronic device (200)) comprises one or more microphones (210), one or more cameras (220), at least one processor (260), and a memory (270) for storing instructions, wherein the instructions are executed by the at least one processor (260) so that the electronic device (200) detects a recording target event while collecting user activity-related data, and in response to the detection of the event, records activity information obtained from at least one of the one or more microphones (210) or the one or more cameras (220) for a specified time, determines whether a content creation condition is satisfied based on the activity-related data, and if the content creation condition is satisfied, determines a content creation item for producing media content, creates media content by combining at least a portion of the recorded activity information based on the determined content creation item, and stores the created media content.

[0119] In one embodiment, the electronic device (200) further includes one or more sensors (230), and the activity-related data may include at least one of voice data obtained by the one or more microphones (210), image data obtained by the one or more cameras (220), and the user's biometric data or location data obtained by the one or more sensors (230).

[0120] In one embodiment, the electronic device (200) further includes a communication circuit (240) that supports a wireless connection with at least one external electronic device, and the instructions are executed by the at least one processor (260) so that when the electronic device (200) is connected to the first external electronic device using the communication circuit (240), the electronic device (200) can obtain the activity-related data from the first external electronic device.

[0121] In one embodiment, the instructions are executed by the at least one processor (260) so that the electronic device (200) determines that the recording target event has occurred when activity-related data exceeding a specified range based on the user's average activity data is detected while collecting the activity-related data.

[0122] In one embodiment, the instructions are executed by the at least one processor (260) so that the electronic device (200) provides the activity-related data to a first artificial intelligence model trained to determine whether the recording target event has occurred, thereby obtaining a determination result regarding whether the event has occurred, and if it is determined that the event has occurred, the activity-related data collected from the time of the event occurrence can be stored as the user's activity information in a first designated area of ​​the memory.

[0123] In one embodiment, the instructions are executed by the at least one processor (260) so that the electronic device (200) provides the record-related data of the first designated area to a second artificial intelligence model trained to determine whether to delete one or more activity information stored in the first designated area, thereby obtaining a determination result regarding whether to delete each activity information, and manages the first designated area by maintaining or deleting the one or more activity information in the first designated area based on the determination result.

[0124] In one embodiment, the instructions are executed by the at least one processor (260) so that the electronic device (200) identifies a user command requesting the creation of media content or a schedule command specifying conditions for the creation of the media content, and provides the activity-related data, the user command, or the schedule command to a third artificial intelligence model trained to determine a content creation item required for the creation of the media content, thereby obtaining the content creation item including at least one of the type of the media content, file format, time of creation, or activity information to be used for the creation of the media content.

[0125] In one embodiment, the instructions may be executed by the at least one processor (260) so that the electronic device (200) provides the content creation item to a fourth artificial intelligence model trained to create media content regarding the user's activity to acquire the media content and store the acquired media content in a second designated area of ​​the memory.

[0126] In one embodiment, the instructions are executed by the at least one processor (260) to enable the electronic device (200) to determine whether one of the one or more persons included in the generated media content is a person not allowed to be recorded, and if the person not allowed to be recorded is identified, to remove the at least one identified person from the media content.

[0127] In one embodiment, the electronic device (200) further includes a display (250), and the instructions are executed by the at least one processor (260) to enable the electronic device (200) to display the generated media content through the display (250).

[0128] A method according to one embodiment may include: detecting a recording target event while collecting user activity-related data; recording activity information obtained using at least one of one or more microphones (210), one or more cameras (220), or one or more sensors (230) for a specified period in response to the detection of the event; determining whether a content creation condition is satisfied based on the activity-related data; determining a content creation item for producing media content if the content creation condition is satisfied; creating media content by combining at least a portion of the recorded activity information based on the determined content creation item; and storing the created media content.

[0129] In one embodiment, the activity-related data may include at least one of voice data obtained from one or more microphones (210), image data obtained from one or more cameras (220), and the user's biometric data or location data obtained from one or more sensors (230).

[0130] In one embodiment, the method may further include the operation of collecting activity-related data from one or more microphones (210), one or more cameras (220), or one or more sensors (230), and / or the operation of obtaining activity-related data from a first external electronic device wirelessly connected to the electronic device.

[0131] In one embodiment, the operation of detecting the recording target event may include determining that the recording target event has occurred when activity data exceeding a specified range based on the user's average activity data is detected while collecting the activity data.

[0132] In one embodiment, the operation of detecting the event to be recorded may include providing the activity-related data to a first artificial intelligence model trained to determine whether the event to be recorded has occurred, thereby obtaining a determination result regarding whether the event has occurred; and, if it is determined that the event has occurred, storing the activity-related data collected from the time of occurrence of the event as the user's activity information in a first designated area of ​​the memory.

[0133] In one embodiment, the method may further include the operation of providing the record-related data of the first designated area to a second artificial intelligence model trained to determine whether to delete one or more activity information stored in the first designated area to obtain a determination result regarding whether to delete each activity information, and the operation of managing the first designated area by maintaining or deleting the one or more activity information in the first designated area based on the determination result.

[0134] In one embodiment, the operation of determining a content creation item for the production of the media content may include: an operation of identifying a user command requesting the production of the media content or a schedule command specifying the conditions for the production of the media content; and an operation of providing the activity-related data, the user command, or the schedule command to a third artificial intelligence model trained to determine a content creation item required for the production of the media content, thereby obtaining the content creation item including at least one of the type of the media content, file format, time of production, or activity information to be used for the production of the media content.

[0135] In one embodiment, the operation of generating the media content may include providing the content generation item to a fourth artificial intelligence model trained to generate media content regarding the user's activity to obtain the media content, and storing the obtained media content in a second designated area of ​​the memory.

[0136] In one embodiment, the method may further include an operation of determining whether one of the one or more persons included in the generated media content is identified as a person not allowed to be recorded, and an operation of removing at least one identified person from the media content if the person not allowed to be recorded is identified.

[0137] In one embodiment, the method may further include the operation of displaying the generated media content through a display (250).

[0138] In one embodiment, programs executable on a computer-readable recording medium are recorded thereon, and the programs may be triggered by the computer to perform the following operations: detecting a recording target event while collecting user activity-related data; recording activity information obtained using at least one of one or more microphones (210), one or more cameras (220), or one or more sensors (230) for a specified period in response to the detection of the event; determining whether a content creation condition is satisfied based on the activity-related data; determining a content creation item for producing media content if the content creation condition is satisfied; creating media content by combining at least a portion of the recorded activity information based on the determined content creation item; and storing the created media content.

[0139] The effects obtainable from the present disclosure are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art to which the present disclosure belongs.

[0140] The electronic device according to the various embodiments disclosed in this document may be of various forms. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a consumer electronics device. The electronic device according to the embodiments of this document is not limited to the devices described above.

[0141] The various embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In this document, phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B and C," and "at least one of A, B, or C" may each include any one of the items listed together in the corresponding phrase, or any combination thereof. Terms such as "first," "second," or "first" or "second" may be used simply to distinguish said components from other said components and do not limit said components in any other aspect (e.g., importance or order). Where any (e.g., 1st) component is referred to as "coupled" or "connected" to another (e.g., 2nd) component, with or without the terms "functionally" or "communicationly," it means that said any component may be connected to said other component directly (e.g., via a wire), wirelessly, or through a third component.

[0142] As used in this document, the term "module" may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0143] Various embodiments of the present document may be implemented as software (e.g., program (140)) comprising one or more instructions stored in a storage medium (e.g., internal memory (136) or external memory (138)) readable by a machine (e.g., electronic device (101)). For example, a processor (e.g., processor (120)) of the machine (e.g., electronic device (101)) may call at least one of the one or more instructions stored in the storage medium and execute it. This enables the machine to be operated to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by the machine may be provided in the form of a non-transitory storage medium. Here, "non-transient" simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily.

[0144] According to one embodiment, the method according to the various embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or an application store (e.g., Play Store). TM It can be distributed online (e.g., downloaded or uploaded) through ) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0145] According to various embodiments, each component (e.g., module or program) of the components described above may include a singular or multiple entities. According to various embodiments, one or more of the components or operations among the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the components of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to the integration. According to various embodiments, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In an electronic device (200), One or more microphones (210); One or more cameras (220); At least one processor (260); and Executed by the above at least one processor (260), the electronic device (200): Detecting events to be recorded while collecting user activity-related data, and In response to the detection of the above event, activity information obtained from at least one of the one or more microphones (210) or the one or more cameras (220) is recorded for a specified period of time, and Based on the data related to the above activity, determine whether the content creation conditions are met, and When the above content creation conditions are satisfied, determine the content creation items for the production of media content, and Based on the above-determined content creation items, media content is created by combining at least a portion of the above-determined recorded activity information, and An electronic device comprising a memory (270) for storing instructions for storing the media content generated above.

2. In Claim 1, It further includes one or more sensors (230), The electronic device, wherein the activity-related data comprises at least one of voice data acquired by one or more microphones (210), image data acquired by one or more cameras (220), and the user's biometric data or location data acquired by one or more sensors (230).

3. In Claim 1, It further includes a communication circuit (240) that supports wireless connection with at least one external electronic device, and The above instructions are executed by the at least one processor (260), and the electronic device (200), An electronic device that obtains activity-related data from the first external electronic device when connected to the first external electronic device using the communication circuit (240).

4. In Claim 1, The above instructions are executed by the at least one processor (260), and the electronic device (200), An electronic device that determines that the recording target event has occurred when activity-related data exceeding a specified range based on the average activity data of the user is detected while collecting the above activity-related data.

5. In Claim 4, The above instructions are executed by the at least one processor (260), and the electronic device (200), The data related to the above activity is provided to a first artificial intelligence model trained to determine whether the above-mentioned event to be recorded has occurred, and a determination result regarding whether the above-mentioned event has occurred is obtained. An electronic device that, when it is determined that the above event has occurred, stores the activity-related data collected from the time of occurrence of the above event as the user's activity information in a first designated area of ​​the memory.

6. In Claim 5, The above instructions are executed by the at least one processor (260), and the electronic device (200), The record-related data of the first designated area is provided to a second artificial intelligence model trained to determine whether to delete one or more activity information stored in the first designated area, thereby obtaining a determination result regarding whether to delete each activity information. An electronic device that manages the first designated area by maintaining or deleting one or more activity information in the first designated area based on the above judgment result.

7. In Claim 1, The above instructions are executed by the at least one processor (260), and the electronic device (200), Identifying a user command requesting the creation of media content or a schedule command specifying the conditions for the creation of said media content, and An electronic device that provides the above activity-related data, the above user command, or the above schedule command to a third artificial intelligence model trained to determine a content creation item necessary for the creation of the above media content, thereby obtaining the content creation item including at least one of the type, file format, creation time, or activity information to be used for the creation of the above media content.

8. In Claim 7, The above instructions are executed by the at least one processor (260), and the electronic device (200), The above content generation item is provided to a fourth artificial intelligence model trained to generate media content regarding the user's activity to obtain the media content, and An electronic device that stores the acquired media content in a second designated area of ​​the memory.

9. In Claim 1, The above instructions are executed by the at least one processor (260), and the electronic device (200), Determining whether one or more persons included in the above-mentioned generated media content are identified as persons whose recording is not permitted, and An electronic device that removes at least one identified person from the media content when a person not permitted to record is identified.

10. In Claim 1, It further includes a display (250), The above instructions are executed by the at least one processor (260), and the electronic device (200), An electronic device that displays the generated media content through the above display (250).

11. Regarding the method, An action of detecting events to be recorded while collecting user activity-related data; In response to the detection of the above event, an operation of recording activity information obtained using at least one of one or more microphones (210), one or more cameras (220), or one or more sensors (230) for a specified period of time; An action of determining whether the content creation conditions are satisfied based on data related to the above activity; If the above content creation conditions are satisfied, an action to determine content creation items for the production of media content; The operation of generating media content by combining at least a portion of the recorded activity information based on the above-determined content generation item; and The operation of saving the media content generated above A method including 12. In Claim 11, The above activity-related data comprises at least one of voice data obtained from one or more microphones (210), image data obtained from one or more cameras (220), and the user's biometric data or location data obtained from one or more sensors (230).

13. In Claim 11, The operation of collecting activity-related data from one or more microphones (210), one or more cameras (220), or one or more sensors (230); and / or A method further comprising the operation of acquiring activity-related data from a first external electronic device wirelessly connected to an electronic device.

14. In Claim 11, The operation of detecting the above-mentioned recording target event is, A method comprising determining that the recording target event has occurred when activity data exceeding a specified range based on the average activity data of the user is detected while collecting the activity data.

15. In Claim 11, The operation of determining content creation items for the production of the above media content is, An operation to identify a user command requesting the creation of the media content or a schedule command specifying the conditions for the creation of the media content; and A method comprising the operation of providing the above activity-related data, the above user command, or the above schedule command to a third artificial intelligence model trained to determine a content creation item necessary for the creation of the above media content, and obtaining the content creation item including at least one of the type, file format, time of creation, or activity information to be used for the creation of the above media content.

Images