An electronic device providing summary information about image and method for controlling same

Abstract

An electronic device for providing summary information about an image and a method for controlling the same are provided. An electronic device according to one embodiment of the present disclosure may include instructions configured to provide summary information including a summary image corresponding to the first time interval of the first image and, while displaying the first image through the display, a user input requesting the provision of summary information corresponding to the first time interval of the first image from a user, based on the acquisition of the user input, context information corresponding to the first time interval of the first image, and using the identified context information, a summary result for the first time interval of the first image through an artificial intelligence learning model of the electronic device, and at least one main frame among a plurality of frames included within the first time interval of the first image, and a summary image corresponding to the at least one main frame and the acquired summary result.

Description

Technology Field

[0001] The present disclosure relates to an electronic device that provides summary information about an image and a method for controlling the same. Background Technology

[0002] Various services and additional functions provided through electronic devices, such as smart TVs, are gradually increasing. To enhance the utility value of these electronic devices and satisfy the needs of diverse users, telecommunications service providers and manufacturers of electronic devices are competitively developing devices to offer various functions and differentiate themselves from competitors. Consequently, the various functions provided through electronic devices are also becoming increasingly sophisticated. means of solving the problem

[0003] According to related technologies for providing summary information about videos, a technique is provided that sets information about text included in the video (e.g., subtitles) as a prompt input to a learning model (e.g., LLM). However, such related technologies generate summary information by setting only information about the text included in the video (e.g., subtitles) as a prompt, resulting in the provision of summary information of relatively low quality to the user, or failing to provide summary information as an appropriate response to various user requests.

[0004] According to one embodiment of the present disclosure, an electronic device may be provided capable of providing summary information that responds to various requests of a user while providing relatively high-quality summary information by identifying context information in a video (e.g., a video being played through an electronic device), generating summary text information and summary scene information based on the identified context information, and synthesizing the generated summary text information and summary scene information to provide to a user as summary information.

[0005] According to one embodiment of the present disclosure, a method for controlling an electronic device is provided that identifies context information in an image (e.g., an image being played through an electronic device), generates summary text information and summary scene information based on the identified context information, and synthesizes the generated summary text information and summary scene information to provide summary information to a user, thereby providing summary information of relatively high quality while also providing summary information that responds to various requests of the user.

[0006] An electronic device according to one embodiment of the present disclosure comprises a display, one or more processors, and a memory for storing instructions, wherein the instructions may include instructions configured to enable the electronic device, when executed individually or collectively by the one or more processors: to display a first image through the display, and while displaying the first image through the display, to obtain user input requesting the provision of summary information corresponding to a first time interval of the first image from a user, based on the acquisition of the user input, to identify context information corresponding to the first time interval of the first image, and using the identified context information, to obtain a summary result during the first time interval of the first image through an artificial intelligence learning model of the electronic device, to identify at least one major frame among a plurality of frames included within the first time interval of the first image, and to provide the summary information including a summary image corresponding to the identified at least one major frame and the obtained summary result.

[0007] A non-transient recording medium according to one embodiment of the present disclosure may include instructions configured to store computer-executable instructions, wherein, when executed individually or collectively by one or more processors of an electronic device, the electronic device may include instructions configured to display a first image through a display of the electronic device, and while displaying the first image through the display, obtain user input requesting the provision of summary information corresponding to a first time interval of the first image from a user, based on the acquisition of the user input, identify context information corresponding to the first time interval of the first image, and using the identified context information, obtain a summary result during the first time interval of the first image through an artificial intelligence learning model of the electronic device, identify at least one major frame among a plurality of frames included within the first time interval of the first image, and provide the summary information including a summary image corresponding to the identified at least one major frame and the obtained summary result.

[0008] A method for controlling an electronic device according to one embodiment of the present disclosure may include: an operation of displaying a first image through a display of the electronic device; an operation of obtaining a user input requesting the provision of summary information corresponding to a first time interval of the first image from a user while displaying the first image through the display; an operation of verifying context information corresponding to the first time interval of the first image based on the acquisition of the user input; an operation of obtaining a summary result during the first time interval of the first image through an artificial intelligence learning model of the electronic device using the verified context information; an operation of identifying at least one main frame among a plurality of frames included within the first time interval of the first image; and an operation of providing the summary information including a summary image corresponding to the identified at least one main frame and the obtained summary result. Brief explanation of the drawing

[0009] FIG. 1 is a block diagram of an electronic device in a network environment according to various embodiments of the present disclosure. FIG. 2 is an exemplary drawing for explaining the function or operation of an electronic device according to one embodiment of the present disclosure that generates and provides summary information. FIG. 3 is an exemplary drawing for explaining the function or operation of an electronic device according to one embodiment of the present disclosure providing a first image including subtitles. FIG. 4 is an exemplary drawing for explaining a function or operation in which an electronic device according to one embodiment of the present disclosure identifies context information of a first image and generates summary text information and summary scene information based on the identified context information to provide summary information. FIG. 5 is an exemplary drawing for explaining a function or operation in which an electronic device according to one embodiment of the present disclosure identifies a key scene corresponding to a key moment. FIG. 6 is an exemplary drawing for explaining a function or operation in which an electronic device according to one embodiment of the present disclosure identifies an active speaker and identifies context information about the active speaker based on the utterance of the active speaker. FIG. 7 is an exemplary drawing for explaining a function or operation in which an electronic device according to one embodiment of the present disclosure identifies information about a designated scene included in a first image. FIG. 8 is an exemplary drawing for illustrating identified context information, summary text information, and summary scene information stored in an electronic device according to one embodiment of the present disclosure. FIG. 9 is an example drawing for explaining a function or operation in which an electronic device according to one embodiment of the present disclosure synthesizes summary text information and summary scene information and provides them to a user as summary information. FIGS. 10a and FIGS. 10b are exemplary drawings for explaining the function or operation of an electronic device according to one embodiment of the present disclosure providing summary information together with a first image. Specific details for implementing the invention

[0010] FIG. 1 is a block diagram of an electronic device (101) in a network environment (100) according to various embodiments.

[0011] Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) through a first network (198) (e.g., a short-range wireless communication network) or with at least one of an electronic device (104) or a server (108) through a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) through a server (108). According to one embodiment, the electronic device (101) may include a processor (120), memory (130), input module (150), sound output module (155), display module (160), audio module (170), sensor module (176), interface (177), connection terminal (178), haptic module (179), camera module (180), power management module (188), battery (189), communication module (190), subscriber identification module (196), or antenna module (197). In some embodiments, at least one of these components (e.g., connection terminal (178)) may be omitted from the electronic device (101), or one or more other components may be added. In some embodiments, some of these components (e.g., sensor module (176), camera module (180), or antenna module (197)) may be integrated into a single component (e.g., display module (160)).

[0012] The processor (120) can control at least one other component (e.g., hardware or software component) of the electronic device (101) connected to the processor (120) by executing software (e.g., program (140)), and can perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor (120) can store commands or data received from other components (e.g., sensor module (176) or communication module (190)) in volatile memory (132), process the commands or data stored in volatile memory (132), and store the resulting data in non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., central processing unit or application processor) or an auxiliary processor (123) that can operate independently or together with it (e.g., graphics processing unit, neural processing unit (NPU), image signal processor, sensor hub processor, or communication processor). For example, if the electronic device (101) includes a main processor (121) and an auxiliary processor (123), the auxiliary processor (123) may be configured to use lower power than the main processor (121) or to be specialized for a designated function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as part thereof.

[0013] The auxiliary processor (123) may control at least some of the functions or states associated with at least one component of the electronic device (101) (e.g., display module (160), sensor module (176), or communication module (190)) on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. According to one embodiment, the auxiliary processor (123) (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module (180) or communication module (190)). According to one embodiment, the auxiliary processor (123) (e.g., neural network processing unit) may include a hardware structure specialized for processing an artificial intelligence learning model. The artificial intelligence learning model may be generated through machine learning. Such learning may be performed, for example, on the electronic device (101) itself where the artificial intelligence learning model is performed, or through a separate server (e.g., server (108)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence learning model may include a plurality of artificial neural network layers.An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. An artificial intelligence learning model may include, in addition to or substantially, a software structure in addition to a hardware structure.

[0014] The memory (130) can store various data used by at least one component of the electronic device (101) (e.g., processor (120) or sensor module (176)). The data may include, for example, input data or output data for software (e.g., program (140)) and related commands. The memory (130) may include volatile memory (132) or non-volatile memory (134).

[0015] The program (140) may be stored as software in memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0016] The input module (150) can receive commands or data to be used for a component of the electronic device (101) (e.g., processor (120)) from outside the electronic device (101) (e.g., user). The input module (150) may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0017] The sound output module (155) can output a sound signal to the outside of the electronic device (101). The sound output module (155) may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as multimedia playback or recording playback. The receiver may be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part thereof.

[0018] The display module (160) can visually provide information to an external (e.g., user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling said device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of the force generated by said touch.

[0019] The audio module (170) can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150) or output sound through the sound output module (155) or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphones) connected directly or wirelessly to the electronic device (101).

[0020] The sensor module (176) can detect the operating state of the electronic device (101) (e.g., power or temperature) or the external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, the sensor module (176) may include, for example, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0021] The interface (177) may support one or more specified protocols that can be used for the electronic device (101) to be connected directly or wirelessly to an external electronic device (e.g., electronic device (102)). According to one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0022] The connection terminal (178) may include a connector through which the electronic device (101) can be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0023] The haptic module (179) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that the user can perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module (179) may include, for example, a motor, a piezoelectric element, or an electric stimulation device.

[0024] The camera module (180) can capture still images and video. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0025] The power management module (188) can manage the power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least part of a power management integrated circuit (PMIC).

[0026] The battery (189) can supply power to at least one component of the electronic device (101). According to one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0027] The communication module (190) can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between an electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may include one or more communication processors that operate independently of the processor (120) (e.g., application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., cellular communication module, short-range wireless communication module, or GNSS (global navigation satellite system) communication module) or a wired communication module (194) (e.g., LAN (local area network) communication module, or power line communication module). The corresponding communication module among these communication modules can communicate with an external electronic device (104) through a first network (198) (e.g., a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data association)) or a second network (199) (e.g., a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can identify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) using subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module (196).

[0028] The wireless communication module (192) can support 5G networks and next-generation communication technologies following 4G networks, for example, new radio access technology. NR access technology can support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and connection of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low-latency communications (URLLC)). The wireless communication module (192) can support a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate, for example. The wireless communication module (192) can support various technologies for securing performance in the high-frequency band, such as beamforming, massive MIMO (multiple-input and multiple-output), full-dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large-scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), external electronic device (e.g., electronic device (104)), or network system (e.g., second network (199)). According to one embodiment, the wireless communication module (192) can support a Peak data rate (e.g., 20 Gbps or more) for realizing eMBB, loss coverage (e.g., 164 dB or less) for realizing mMTC, or U-plane latency (e.g., downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less) for realizing URLLC.

[0029] An antenna module (197) can transmit a signal or power to or from an external source (e.g., an external electronic device). According to one embodiment, the antenna module (197) may include an antenna comprising a radiator made of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as a first network (198) or a second network (199), may be selected from the plurality of antennas, for example, by a communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, other components (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module (197).

[0030] According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the printed circuit board and capable of supporting a specified high frequency band (e.g., mmWave band), and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top surface or side surface) of the printed circuit board and capable of transmitting or receiving a signal of the specified high frequency band.

[0031] At least some of the above components can be connected to each other via a communication method between peripheral devices (e.g., bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)) and exchange signals (e.g., commands or data) with each other.

[0032] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) through a server (108) connected to a second network (199). Each of the external electronic devices (102, or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations performed on the electronic device (101) may be performed on one or more of the external electronic devices (102, 104, or 108). For example, if the electronic device (101) needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device (101) may request one or more external electronic devices to perform at least part of the function or service instead of performing the function or service itself or additionally. One or more external electronic devices that receive the above request may execute at least part of the requested function or service, or additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may provide the result as is or additionally processed as at least part of the response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device (101) may provide ultra-low latency services using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server using machine learning and / or neural networks. According to one embodiment, the external electronic device (104) or the server (108) may be included within a second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0033] FIG. 2 is an exemplary drawing for explaining a function or operation in which an electronic device (101) according to one embodiment of the present disclosure generates and provides summary information (e.g., summary information (1010) of 10b). FIG. 3 is an exemplary drawing for explaining a function or operation in which an electronic device (101) according to one embodiment of the present disclosure provides a first image (310) including subtitles (312).

[0034] Referring to FIG. 2, an electronic device (101) according to one embodiment of the present disclosure may display a first image (310) through a display (e.g., a display module (160) of FIG. 1) in operation 210. An electronic device (101) according to one embodiment of the present disclosure may provide a first image including subtitles (312) through the display module (160). A first image (310) according to one embodiment of the present disclosure may include a plurality of video frames (e.g., video frames (510) of FIG. 5). A first image (310) according to one embodiment of the present disclosure may include time information (e.g., time code and / or timestamp information) corresponding to each of the plurality of video frames (510). A first image (310) according to one embodiment of the present disclosure may include descriptive information for at least one frame among the plurality of video frames (510). A first image (310) according to one embodiment of the present disclosure may include audio information. A first image (310) according to one embodiment of the present disclosure may include an image that is at least temporarily stored in an electronic device (101) or an image transmitted from an external electronic device (e.g., a server) connected to the electronic device (101) so as to be operable.

[0035] An electronic device (101) according to one embodiment of the present disclosure may, in operation 220, obtain user input requesting the provision of summary information corresponding to a first time interval of the first image (310) (e.g., 15 minutes prior to the current time) while displaying a first image (310) through a display (e.g., the display module (160) of FIG. 1). An electronic device (101) according to one embodiment of the present disclosure may obtain user utterances such as, for example, “Tell me what was interesting during the last 15 minutes.” An electronic device (101) according to one embodiment of the present disclosure may recognize voice. According to one embodiment, the electronic device (101) may recognize the user’s voice and interpret the intent of the utterance included in the recognized voice by receiving a voice signal that is an analog signal through an input module (150) (e.g., a microphone). According to one embodiment, an electronic device (101) can recognize audio included in a video and interpret the intent of the voice included in the recognized audio. The electronic device (101) can convert the voice portion into computer-readable text using an Automatic Speech Recognition (ASR) model. It can obtain the speech intent by interpreting the converted text using a Natural Language Understanding (NLU) model. Here, the ASR model or the NLU model may be an artificial intelligence learning model. The artificial intelligence learning model may be processed by a processor (120) (e.g., an artificial intelligence dedicated processor) designed with a hardware structure specialized for processing the artificial intelligence learning model. The artificial intelligence learning model may be created through learning.Linguistic understanding is a technology that recognizes, applies, and processes human language / characters, and may include natural language processing, machine translation, dialogue systems, question answering, speech recognition / synthesis, etc. An electronic device (101) according to one embodiment of the present disclosure may represent acquired user utterances in the form of dense vectors. In other words, an electronic device (101) according to one embodiment of the present disclosure may process user utterances through a word embedding method. An electronic device (101) according to one embodiment of the present disclosure may acquire text information, such as "Tell me what was interesting during the last 15 minutes," through a text input window instead of user utterances. When an electronic device (101) according to one embodiment of the present disclosure acquires text information instead of user utterances, it may identify the intention of the user who entered the text based on the acquired text information.

[0036] An electronic device (101) according to one embodiment of the present disclosure can, in operation 230, determine context information corresponding to a first time interval of a first image (310) based on the acquisition of user input according to operation 220. The context information according to one embodiment of the present disclosure may include information about a speaker of a designated scene, information about subtitles (312) included in the first image (310), information about the speaker's utterance, screen information corresponding to a plurality of frames included in the first image (310) (e.g., description of a designated frame), information about a time code of the first image (310), and / or information about audio included in the first image (310).

[0037] An electronic device (101) according to one embodiment of the present disclosure can identify an active speaker (e.g., a speaker who is speaking) included in a designated scene. FIG. 6 is an exemplary drawing for explaining a function or operation in which an electronic device (101) according to one embodiment of the present disclosure identifies an active speaker and identifies context information (e.g., the emotions of the active speaker) about the active speaker (e.g., the first speaker (610)) based on the utterance of the active speaker (e.g., the first speaker (610)).

[0038] Referring to FIG. 6, an electronic device (101) according to one embodiment of the present disclosure can identify that audio data corresponding to a designated frame does not exist in a designated frame (e.g., a first frame (600a) included within a first time interval). In this case, the electronic device (101) according to the present disclosure can recognize an image (e.g., identify a scene) and determine whether an active speaker is present in the scene. To this end, the electronic device (101) according to the present disclosure can recognize a frame (e.g., an image). According to one embodiment, the electronic device (101) can obtain output data that recognizes an image by using the image data as input data for an artificial intelligence learning model. An artificial intelligence learning model according to one embodiment of the present disclosure can be generated through learning. An artificial intelligence learning model according to one embodiment of the present disclosure can recognize (e.g., understand) an image based on a visual understanding of the image. Visual understanding according to one embodiment of the present disclosure is a technology that recognizes and processes objects like human vision, and may include object recognition, object tracking, image retrieval, human reconnition, scene recognition, spatial understanding (3D reconstruction / localization), image enhancement, etc.

[0039] An electronic device (101) according to one embodiment of the present disclosure can identify an active speaker using audio data and image data corresponding to a designated frame (e.g., a second frame (600b) included within a first time interval). An electronic device (101) according to one embodiment of the present disclosure can identify a second speaker (620) as an active speaker in a designated frame (e.g., a second frame (600b) included within a first time interval). An electronic device (101) according to one embodiment of the present disclosure can identify the second speaker (620) as an active speaker based, for example, on the movement of the speaker's mouth shape included in an image and audio data. An electronic device (101) according to one embodiment of the present disclosure can identify (e.g., infer) contextual information about the active speaker, such as the gender, emotion, age, etc. of the active speaker (e.g., the second speaker (620)), based on visual understanding of the image.

[0040] An electronic device (101) according to one embodiment of the present disclosure can identify an active speaker using audio data and image data corresponding to a designated frame (e.g., a third frame (600c) included within a first time interval). An electronic device (101) according to one embodiment of the present disclosure can identify a first speaker (610) as an active speaker in a designated frame (e.g., a third frame (600c) included within a first time interval). An electronic device (101) according to one embodiment of the present disclosure can identify the first speaker (610) as an active speaker based on, for example, the movement of the speaker's mouth shape included in an image and audio data. An electronic device (101) according to one embodiment of the present disclosure can identify (e.g., infer) contextual information about the active speaker (e.g., the gender, emotion, age, etc.) of the active speaker (e.g., the first speaker (610)) based on visual understanding of the image.

[0041] An electronic device (101) according to one embodiment of the present disclosure can identify an active speaker using audio data corresponding to a designated frame (e.g., a fourth frame (600d) included within a first time interval). An electronic device (101) according to one embodiment of the present disclosure can identify a second speaker (620) as an active speaker in a designated frame (e.g., a fourth frame (600d) included within a first time interval). An electronic device (101) according to one embodiment of the present disclosure can identify the second speaker (620) as an active speaker based, for example, attributes of the audio data (e.g., intonation) and / or information about a previous frame (e.g., information about who was the active speaker). An electronic device (101) according to one embodiment of the present disclosure can identify a time code corresponding to at least one designated frame (e.g., a first frame (600a), a second frame (600b), a third frame (600c), and / or a fourth frame (600d) included within a first time interval).

[0042] An electronic device (101) according to one embodiment of the present disclosure may recognize voice and / or subtitles included in the first image (310) as context information for the first image (310). The recognition of voice and / or subtitles according to one embodiment of the present disclosure may be performed, for example, by an artificial intelligence learning model. An electronic device (101) according to one embodiment of the present disclosure may obtain recognized text and / or a time code corresponding to the recognized text as a result of the recognition of voice and / or subtitles included in the first image (310).

[0043] According to one embodiment, an artificial intelligence learning model according to one embodiment of the present disclosure may include a large language model (LLM). According to one embodiment, the artificial intelligence learning model may be stored in the memory (130) of an electronic device (101) and may also be stored on an external server (e.g., the server (108) of FIG. 1). An LLM according to one embodiment of the present disclosure may refer to a language model based on an artificial neural network that has learned a large amount of text data through prior training. An LLM according to one embodiment of the present disclosure may include far more parameters (e.g., more than 10 billion) than a conventional general language model. An LLM may use a transformer artificial neural network structure based on an attention mechanism. An attention mechanism according to one embodiment of the present disclosure is a technology that helps an artificial intelligence learning model focus on important parts within input data. The attention mechanism can be utilized for output data prediction by predicting the extent to which a portion of time-series input data (e.g., input data such as voice or video, or input data for a specific layer of a neural network) contributes to the intermediate or final output of the neural network. While Recurrent Neural Network (RNN) structures, which process each element of a sequence sequentially, suffer from degraded prediction performance when there is information dependency over long time-series distances, the attention mechanism can account for information dependency over long time-series distances by controlling the degree of weight concentration (attention) within the context of the input data as a whole (or part thereof). The transformer can be composed of an encoder-decoder structure.The encoder processes input data to output compressed information (e.g., contextual representation), and the decoder processes the compressed information to output data in token units. Each of the encoder and decoder may include an independent attention network and a cross-attention network connecting the encoder and decoder.

[0044] For example, LLM training according to one embodiment of the present disclosure may include pre-training and / or fine-tuning. Pre-training according to one embodiment of the present disclosure is a process of enabling the LLM to acquire general language knowledge using a large amount of text data, and may include, for example, self-supervised learning that predicts the next word using the previous sequence of words in a sequence of text. Fine-tuning according to one embodiment of the present disclosure is a process of training the LLM to be suitable for a specific domain (e.g., chatbot, translation, summarization, Q&A) or task, and the LLM may be further supervised (or adaptive) using a dataset suitable for the domain purpose based on a pre-trained model. The LLM according to one embodiment of the present disclosure may perform tasks using text input containing natural language called a prompt. For example, fine-tuning may be omitted during LLM training. The prompt to be input into the LLM can be controlled to increase the performance of the task desired by the user. In a manner similar to in-context learning or zero-shot / few-shot learning, examples of tasks and / or guides for performing tasks can be additionally provided to the prompt.

[0045] An LLM according to one embodiment of the present disclosure may additionally receive image information in addition to text. The image information according to one embodiment of the present disclosure may be converted into text through a separate pre-conversion (e.g., image recognition, scene recognition) and may include this in a prompt to generate a response. As another example, an input image may be converted into image embeddings aligned with text through an image encoder, and a response may be generated using a separately trained model (e.g., a large multimodal model) utilizing text embeddings corresponding to the input text.

[0046] The expression 'LLM' may refer to the language neural network model itself, but it may also refer to a model of an LLM-based application (e.g., chatbot, translation, summarization, text classification, sentence generation). For example, an LLM-based chatbot or LLM-based translator such as ChatGPT may also be referred to as 'LLM'. 'LLM' according to one embodiment of the present disclosure may include an inference engine using an LLM neural network model. For example, "inputting an input prompt into the LLM" may mean "inputting an input prompt into an LLM-based inference engine." For example, "the output of the LLM for the input prompt" may refer to the output information of the last neural network layer of the LLM (or output information modified through additional processing) obtained when the input prompt is input into an LLM-based inference engine.

[0047] Functions related to artificial intelligence according to the present disclosure may be operated through a processor (120) and a memory (130). The processor (120) may be composed of one or more processors (120). In this case, the one or more processors (120) may be general-purpose processors such as a CPU, AP, DSP (Digital Signal Processor), graphics-dedicated processors such as a GPU, VPU (Vision Processing Unit), or artificial intelligence-dedicated processors such as an NPU. The one or more processors (120) control input data to be processed according to predefined operation rules or artificial intelligence learning models stored in the memory (130). Alternatively, if the one or more processors (120) are artificial intelligence-dedicated processors, the artificial intelligence-dedicated processors may be designed with a hardware structure specialized for processing a specific artificial intelligence learning model.

[0048] An electronic device (101) according to one embodiment of the present disclosure may recognize a frame (e.g., image and / or scene) included in the first image (310) as context information for the first image (310). An electronic device (101) according to one embodiment of the present disclosure may identify pre-stored descriptive information corresponding to the frame (e.g., image and / or scene) included in the first image (310) and recognize the features of the frame (e.g., image and / or scene) based on the identified descriptive information. Alternatively, an electronic device (101) according to one embodiment of the present disclosure may recognize (e.g., understand) the image based on a visual understanding of the image. FIG. 7 is an exemplary drawing for explaining a function or operation in which an electronic device (101) according to one embodiment of the present disclosure identifies information about a designated scene included in the first image (310).

[0049] Referring to FIG. 7, an electronic device (101) according to one embodiment of the present disclosure may understand the features of a frame (e.g., image and / or scene) based on a visual understanding of an image and generate a description corresponding to the frame (e.g., image and / or scene). For example, an electronic device (101) according to one embodiment of the present disclosure may identify, as illustrated in FIG. 7, "a man wearing a suit is in a news feed" as a description of a designated frame. Or, an electronic device (101) according to one embodiment of the present disclosure may identify, as a description of a designated frame, "a man wearing a suit and tie is standing next to a woman wearing a suit and tie." An electronic device (101) according to one embodiment of the present disclosure may identify attributes of an image (e.g., whether a designated image is an image related to news content, an image related to sports content, or an image related to music content) as a result of a visual understanding of an image. An electronic device (101) according to one embodiment of the present disclosure can identify a time code corresponding to a designated frame (e.g., a designated frame in which visual understanding is performed) and identify and / or store information about the identified time code as context information.

[0050] Returning to FIG. 2, an electronic device (101) according to one embodiment of the present disclosure can identify information about audio (e.g., background sound other than voice) included in the first image (310) as context information for the first image (310). An electronic device (101) according to one embodiment of the present disclosure can identify characteristics of the audio (e.g., background sound other than voice) based on a description of the audio stored in advance corresponding to the audio data. Alternatively, an electronic device (101) according to one embodiment of the present disclosure may identify characteristics of the audio (e.g., background sound other than voice) using an artificial intelligence learning model. An electronic device (101) according to one embodiment of the present disclosure can identify a time code corresponding to a designated frame (e.g., a frame containing designated audio data) and identify and / or store information about the identified time code as context information.

[0051] An electronic device (101) according to one embodiment of the present disclosure can obtain a summary result of a first time interval (e.g., the past 15 minutes) of a first video (310) through an artificial intelligence learning model using context information identified according to operation 230 in operation 240. An electronic device (101) according to one embodiment of the present disclosure can generate a prompt including the acquired context information. For example, an electronic device (101) according to one embodiment of the present disclosure can generate a prompt set to "summarize the scene of the past 15 minutes using context information" and input the generated prompt into an artificial intelligence learning model. An electronic device (101) according to one embodiment of the present disclosure can input the identified context information into an artificial intelligence learning model. An electronic device (101) according to one embodiment of the present disclosure can obtain a summary result (e.g., "a fire broke out in the building while the men were talking") in response to the input prompt.

[0052] An electronic device (101) according to one embodiment of the present disclosure can, in operation 250, identify at least one major frame (e.g., the first major frame (510a) of FIG. 5, the second major frame (510b) of FIG. 5) among a plurality of frames included within a first time interval (e.g., the past 15 minutes) of a first image (310) by using context information identified according to operation 240. FIG. 5 is an example drawing for explaining a function or operation in which an electronic device (101) according to one embodiment of the present disclosure identifies a major scene corresponding to a major moment (e.g., as a major frame, the first major frame (510a) of FIG. 5, the second major frame (510b) of FIG. 5).

[0053] Referring to FIG. 5, an electronic device (101) according to one embodiment of the present disclosure may determine an importance coefficient for each of a plurality of frames included within a first time interval (e.g., during the last 15 minutes) of a first image (310) in order to select at least one major frame. An electronic device (101) according to one embodiment of the present disclosure may determine an importance coefficient for each of the plurality of frames using identified context information (e.g., scene importance identified based on the active speaker's emotional information and / or scene description). According to one embodiment of the present disclosure, the scene importance when the active speaker's emotional state is a first state (e.g., angry state) may be higher than the scene importance when the active speaker's emotional state is a second state (e.g., calm state). According to one embodiment of the present disclosure, information regarding the active speaker's emotional state may be identified or obtained in the form of a continuous graph over time. An electronic device (101) according to one embodiment of the present disclosure may set information regarding a scene description as an input value for an artificial intelligence learning model and obtain information regarding the scene importance of a designated scene as an output value for the input value. For example, an artificial intelligence learning model according to one embodiment of the present disclosure may be trained to set the importance relatively high as a result of learning when a designated word is included in the scene description. An artificial intelligence learning model according to one embodiment of the present disclosure may output information regarding the scene importance of a designated scene based on the result of such learning. Alternatively, according to one embodiment of the present disclosure, information regarding the scene importance may be included in the scene description. An electronic device (101) according to one embodiment of the present disclosure may identify the importance of a designated scene using information regarding the scene importance included in the scene description.

[0054] An electronic device (101) according to one embodiment of the present disclosure inputs identified context information (e.g., scene importance identified based on the active speaker's emotion information and scene description) along with a generated prompt (e.g., a prompt that causes the artificial intelligence learning model to "determine an importance coefficient") to an artificial intelligence learning model, and can determine an importance coefficient based on the result of the input. An artificial intelligence learning model according to one embodiment of the present disclosure may be configured to output an importance coefficient as a result of the input of the generated prompt (e.g., a prompt that causes the artificial intelligence learning model to "determine an importance coefficient"). An artificial intelligence learning model according to one embodiment of the present disclosure may determine and output an importance coefficient using a formula configured to calculate the result of learning and / or the importance coefficient. Alternatively, an electronic device (101) according to one embodiment of the present disclosure may determine an importance coefficient using a lookup table in which the relationship between context information (e.g., scene importance identified based on the active speaker's emotion information and / or scene description) and the importance coefficient is defined. Alternatively, an electronic device (101) according to one embodiment of the present disclosure may determine an importance coefficient for each frame using information regarding an importance coefficient that is predetermined for each frame. In this case, context information according to one embodiment of the present disclosure may not be used to identify or select a major scene (e.g., as a major frame, the first major frame (510a) of FIG. 5, the second major frame (510b) of FIG. 5). An electronic device (101) according to one embodiment of the present disclosure (e.g., processor (120)) may identify a major scene corresponding to a major moment (e.g., as a major frame, the first major frame (510a) of FIG. 5, the second major frame (510b) of FIG. 5) based on whether the determined importance coefficient satisfies a specified condition (e.g., when the importance coefficient value is greater than or equal to a specified threshold value).

[0055] In the above-described embodiment, an example in which an artificial intelligence learning model outputs an importance coefficient has been described by way of example; however, according to other embodiments of the present disclosure, an artificial intelligence learning model may be controlled so that a key scene (e.g., as a key frame, the first key frame (510a) of FIG. 5, the second key frame (510b) of FIG. 5) is identified by the artificial intelligence learning model. For example, a generated prompt according to one embodiment of the present disclosure may include a prompt that causes the artificial intelligence learning model to “select a key scene.” In this case, the artificial intelligence learning model according to one embodiment of the present disclosure may be configured to output a key scene corresponding to a key moment (e.g., as a key frame, the first key frame (510a) of FIG. 5, the second key frame (510b) of FIG. 5) as a result of the input of the generated prompt (e.g., the prompt that causes the artificial intelligence learning model to “select a key scene)). An artificial intelligence learning model according to one embodiment of the present disclosure can determine an importance coefficient for each of a plurality of frames included within a first time interval (e.g., during the last 15 minutes) of a first image (310), and identify a major scene corresponding to a major moment (e.g., as a major frame, the first major frame (510a) of FIG. 5, the second major frame (510b) of FIG. 5) based on whether the determined importance coefficient satisfies a specified condition (e.g., when the importance coefficient value is greater than or equal to a specified threshold value).

[0056] Returning to FIG. 2, an electronic device (101) according to one embodiment of the present disclosure may, in operation 260, provide summary information (e.g., summary information of FIG. 10b (1010)) comprising a summary image (e.g., summary image of FIG. 10b (1010a)) corresponding to at least one identified major frame (e.g., first major frame (510a) of FIG. 5, second major frame (510b) of FIG. 5)) and an acquired summary result (e.g., summary result of FIG. 10b (1010b)). An electronic device (101) according to one embodiment of the present disclosure can synthesize summary information (e.g., summary information of FIG. 10b (1010)) comprising a summary image (e.g., summary image of FIG. 10b (1010a)) corresponding to at least one identified main frame (e.g., first main frame (510a) of FIG. 5, second main frame (510b) of FIG. 5) and an acquired summary result (e.g., summary result of FIG. 10b (1010b)). FIG. 9 is an example drawing for explaining a function or operation in which an electronic device (101) according to one embodiment of the present disclosure synthesizes summary text information (e.g., summary result of FIG. 10b (1010b)) and summary scene information (e.g., summary image of FIG. 10b (1010a)) and provides summary information (e.g., summary information of FIG. 10b (1010)) to a user.

[0057] Referring to FIG. 9 to describe a synthesis according to one embodiment of the present disclosure, an electronic device (101) according to one embodiment of the present disclosure can identify at least one major frame (e.g., the first major frame (510a) of FIG. 5, the second major frame (510b) of FIG. 5) among a plurality of frames included within a first time interval of the first image (310) in operation 910.

[0058] An electronic device (101) according to one embodiment of the present disclosure may, in operation 920, process at least one identified major frame (e.g., the first major frame (510a) of FIG. 5, the second major frame (510b) of FIG. 5). For example, an electronic device (101) according to one embodiment of the present disclosure may merge and / or stitch the frames so that at least one identified major frame (e.g., the first major frame (510a) of FIG. 5, the second major frame (510b) of FIG. 5) is naturally connected, and / or align at least one major frame (e.g., the first major frame (510a) of FIG. 5, the second major frame (510b) of FIG. 5) in chronological order based on a time code corresponding to each frame. Alternatively, an electronic device (101) according to one embodiment of the present disclosure may provide additional visual effects and / or additional auditory effects to at least one identified main frame (e.g., the first main frame (510a) of FIG. 5, the second main frame (510b) of FIG. 5).

[0059] An electronic device (101) according to one embodiment of the present disclosure can, in operation 930, synthesize a summary result for a plurality of frames included within a first time interval (e.g., summary result (1010b) of FIG. 10b) generated by an artificial intelligence learning model, and at least one processed main frame (e.g., first main frame (510a) of FIG. 5, second main frame (510b) of FIG. 5). An electronic device (101) according to one embodiment of the present disclosure can synthesize a summary result for a plurality of frames included in a first time interval (e.g., a summary result (1010b) of FIG. 10b)) and at least one processed main frame (e.g., a first main frame (510a) of FIG. 5, a second main frame (510b) of FIG. 5) so that a summary result for a plurality of frames included in a first time interval (e.g., a summary result (1010b) of FIG. 10b)) is included in at least one processed main frame (e.g., a first main frame (510a) of FIG. 5, a second main frame (510b) of FIG. 5). Alternatively, an electronic device (101) according to one embodiment of the present disclosure may group a summary result for a plurality of frames included in a first time interval (e.g., a summary result (1010b) of FIG. 10b) and at least one processed major frame (e.g., a first major frame (510a) of FIG. 5, a second major frame (510b) of FIG. 5)) so that the summary result for a plurality of frames included in a first time interval (e.g., a summary result (1010b) of FIG. 10b) and at least one processed major frame (e.g., a first major frame (510a) of FIG. 5, a second major frame (510b) of FIG. 5)) can be processed as one object.Alternatively, if a summary result for each of at least one main frame (e.g., the first main frame (510a) of FIG. 5, the second main frame (510b) of FIG. 5) is configured to be generated through an artificial intelligence learning model via an electronic device (101) according to one embodiment of the present disclosure, a summary result for a plurality of frames included within a first time interval (e.g., the summary result (1010b) of FIG. 10b)) and at least one processed main frame (e.g., the first main frame (510a) of FIG. 5, the second main frame (510b) of FIG. 5)) may be synthesized so that the at least one main frame (e.g., the first main frame (510a) of FIG. 5, the second main frame (510b) of FIG. 5) and the generated summary result are matched with each other.

[0060] An electronic device (101) according to one embodiment of the present disclosure may, in operation 940, provide a synthesized summary result and at least one main frame to a user as summary information (e.g., summary information (1010) of FIG. 10b). FIG. 10a and FIG. 10b are exemplary drawings for explaining a function or operation in which an electronic device (101) according to one embodiment of the present disclosure provides summary information (1010) together with a first image (310). Referring to FIG. 10a and FIG. 10b, the electronic device (101) according to the embodiment may output summary information (1010) together with a first image (310) currently being played through a display module (160). Alternatively, an electronic device (101) according to one embodiment of the present disclosure may stop the output of a first image (310) that is being played and output the first image (310) that has been stopped from outputting, or only summary information (1010) through the electronic device (101).

[0061] FIG. 4 is an example drawing for explaining a function or operation in which an electronic device (101) according to one embodiment of the present disclosure identifies context information of a first image (310), and generates summary text information (e.g., summary result (1010b) of FIG. 10b) and summary scene information (e.g., summary image (1010a) of FIG. 10b) based on the identified context information to provide summary information (1010).

[0062] Referring to FIG. 4, an electronic device (101) according to one embodiment of the present disclosure may include an interaction module (410), a context identification module (420), an artificial intelligence learning model (430), and / or a summary information generation module (440). At least some of the functions or operations performed by the interaction module (410), the context identification module (420), the artificial intelligence learning model (430), and / or the summary information generation module (440) according to one embodiment of the present disclosure may be performed by a processor (120). Alternatively, at least some of the functions or operations performed by the interaction module (410), the context identification module (420), the artificial intelligence learning model (430), and / or the summary information generation module (440) according to one embodiment of the present disclosure may be performed by an external electronic device (e.g., a server) connected to communicate with the electronic device (101), and the result of the performance may be transmitted to the electronic device (101).

[0063] An interaction module (410) according to one embodiment of the present disclosure may receive commands or data to be used for a component (e.g., processor (120)) of the electronic device (101) from outside the electronic device (101) (e.g., user). An input module (150) may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0064] A context identification module (420) according to one embodiment of the present disclosure may be configured to verify and / or identify context information of a first image (310). Context information according to one embodiment of the present disclosure may include information about a speaker in a designated scene, information about subtitles (312) included in the first image (310), information about the speaker's utterance, screen information corresponding to a plurality of frames included in the first image (310) (e.g., description of a designated frame), information about the time code of the first image (310), and / or information about audio included in the first image (310).

[0065] An artificial intelligence learning model (430) according to one embodiment of the present disclosure may include a large language model (LLM). According to one embodiment, the artificial intelligence learning model may be stored in the memory (130) of an electronic device (101) and may also be stored on an external server (e.g., the server (108) of FIG. 1). An LLM according to one embodiment of the present disclosure may refer to a language model based on an artificial neural network that has learned a large amount of text data through prior learning.

[0066] A summary information generation module (440) according to one embodiment of the present disclosure can process at least one identified major frame (e.g., the first major frame (510a) of FIG. 5, the second major frame (510b) of FIG. 5). Additionally, a summary information generation module (440) according to one embodiment of the present disclosure can synthesize a summary result for a plurality of frames included within a first time interval (e.g., the summary result (1010b) of FIG. 10b)) generated by an artificial intelligence learning model with at least one processed major frame (e.g., the first major frame (510a) of FIG. 5, the second major frame (510b) of FIG. 5).

[0067] FIG. 8 is an example drawing for illustrating identified context information, summary text information, and summary scene information stored in an electronic device (101) according to one embodiment of the present disclosure.

[0068] Referring to FIG. 8, an electronic device (101) according to one embodiment of the present disclosure may store, at least temporarily, a database (e.g., memory (130)) a plurality of frames included in a first image (310), time codes corresponding to each of the plurality of frames, information about a speaker included in the scene, information about text recognized in subtitles (312) and / or audio, and / or summary results for at least some of the plurality of frames.

[0069] User input according to one embodiment of the present disclosure may include various user utterances, for example, “Tell me what happened in this show during the last 20 minutes,” “Give me a list of keywords presented in the second part of the lecture,” and / or “Summarize the content of the previous part of this movie.” Depending on such various user utterances, the electronic device (101) according to one embodiment of the present disclosure may provide the user with the video of the show during the last 20 minutes and the summary results as summary information (1010). Alternatively, the electronic device (101) according to one embodiment of the present disclosure may provide the user with a list of at least one keyword presented by the instructor in the previous lecture video or identified by an artificial intelligence learning model as summary information (1010). Alternatively, an electronic device (101) according to one embodiment of the present disclosure may generate summary information (1010) of the previous part of a movie to be played, and output a guidance message to guide the user to watch the summary information (1010) of the previous part before playing the movie to be played. An electronic device (101) according to one embodiment of the present disclosure may output the summary information (1010) of the previous part based on the acquisition of user input regarding the guidance message.

[0070] An electronic device according to one embodiment of the present disclosure (e.g., the electronic device (101) of FIG. 1) comprises a display (e.g., the display module (160) of FIG. 1), one or more processors (e.g., the processor (120) of FIG. 1), and a memory (e.g., the memory (130) of FIG. 1) for storing instructions, wherein, when the instructions are executed individually or collectively by the one or more processors, the electronic device: displays a first image (e.g., the first image (310) of FIG. 3) through the display; while displaying the first image through the display, obtains user input requesting the provision of summary information corresponding to a first time interval of the first image from a user; based on the acquisition of the user input, identifies context information corresponding to the first time interval of the first image; using the identified context information, obtains a summary result for the first time interval of the first image through an artificial intelligence learning model of the electronic device, and within the first time interval of the first image It may include instructions configured to identify at least one main frame among a plurality of included frames, and to provide summary information (e.g., summary information (1010) of FIG. 10b) including a summary image corresponding to the identified at least one main frame and the acquired summary result.

[0071] A non-transient recording medium according to one embodiment of the present disclosure may include instructions configured to store computer-executable instructions, wherein, when executed individually or collectively by one or more processors of an electronic device, the electronic device may include instructions configured to display a first image through a display of the electronic device, and while displaying the first image through the display, obtain user input requesting the provision of summary information corresponding to a first time interval of the first image from a user, based on the acquisition of the user input, identify context information corresponding to the first time interval of the first image, and using the identified context information, obtain a summary result during the first time interval of the first image through an artificial intelligence learning model of the electronic device, identify at least one major frame among a plurality of frames included within the first time interval of the first image, and provide the summary information including a summary image corresponding to the identified at least one major frame and the obtained summary result.

[0072] A method for controlling an electronic device according to one embodiment of the present disclosure may include: an operation of displaying a first image through a display of the electronic device; an operation of obtaining a user input requesting the provision of summary information corresponding to a first time interval of the first image from a user while displaying the first image through the display; an operation of verifying context information corresponding to the first time interval of the first image based on the acquisition of the user input; an operation of obtaining a summary result during the first time interval of the first image through an artificial intelligence learning model of the electronic device using the verified context information; an operation of identifying at least one main frame among a plurality of frames included within the first time interval of the first image; and an operation of providing the summary information including a summary image corresponding to the identified at least one main frame and the obtained summary result.

[0073] The electronic device according to the various embodiments disclosed in this document may be of various forms. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a consumer electronics device. The electronic device according to the embodiments of this document is not limited to the devices described above.

[0074] The various embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In this document, phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B and C," and "at least one of A, B, or C" may each include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used simply to distinguish said components from other said components and do not limit said components in any other aspect (e.g., importance or order). Where any (e.g., 1st) component is referred to as “coupled” or “connected” to another (e.g., 2nd) component, with or without the terms “functionally” or “communicationly,” it means that said any component may be connected to said other component directly (e.g., via a wire), wirelessly, or through a third component.

[0075] The term “module” as used in the various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0076] Various embodiments of the present document may be implemented as software (e.g., program (140)) comprising one or more instructions stored in a storage medium (e.g., internal memory (136) or external memory (138)) readable by a machine (e.g., electronic device (101)). For example, a processor of the machine (e.g., electronic device (101)) may call at least one of the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by the machine may be provided in the form of a non-transitory storage medium. Here, "non-transient" simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily.

[0077] According to one embodiment, the method according to the various embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or distributed online (e.g., download or upload) through an application store (e.g., Play Store™) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0078] According to one embodiment of this document, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to one embodiment of this document, one or more of the components or operations among the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to the integration. According to one embodiment of this document, operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

Claim 1 An electronic device comprising: a display; one or more processors; and a memory for storing instructions, wherein the instructions, when executed individually or collectively by the one or more processors, cause the electronic device to: display a first image through the display; obtain user input requesting the provision of summary information corresponding to a first time interval of the first image from a user while the first image is displayed through the display; based on the acquisition of the user input, identify context information corresponding to the first time interval of the first image; and using the identified context information: obtain a summary result during the first time interval of the first image through an artificial intelligence learning model of the electronic device; identify at least one major frame among a plurality of frames included within the first time interval of the first image; and provide the summary information including a summary image corresponding to the identified at least one major frame and the obtained summary result. Claim 2 An electronic device according to claim 1, wherein the instructions further include instructions configured to enable the electronic device to identify the at least one main frame by further utilizing the acquired summary result when executed individually or collectively by the one or more processors. Claim 3 An electronic device according to claim 1 or 2, wherein the context information comprises information about a speaker, information about subtitles included in the first image, information about the speaker's utterance, screen information corresponding to a plurality of frames included in the first image, information about the time code of the first image, and / or information about audio included in the first image. Claim 4 An electronic device according to any one of claims 1 to 3, wherein the instructions further comprise instructions configured to cause the electronic device to display the summary information together with the first image through the display when executed individually or collectively by the one or more processors. Claim 5 An electronic device according to any one of claims 1 to 4, wherein the instructions further comprise, when executed individually or collectively by the one or more processors, an instruction configured to identify the at least one major frame among the plurality of frames based on an importance factor (IF) determined based on information regarding the speaker's emotion identified based on the speaker's utterance and information regarding the importance of the scene corresponding to the plurality of frames. Claim 6 An electronic device according to any one of claims 1 to 5, wherein the instructions further comprise instructions configured such that, when executed individually or collectively by the one or more processors, the electronic device obtains a description of a screen corresponding to a plurality of frames included in the first image using the artificial intelligence learning model, and identifies the obtained description of the screen as the screen information. Claim 7 An electronic device according to any one of claims 1 to 6, wherein the instructions further comprise, when executed individually or collectively by the one or more processors, an instruction configured to synthesize the summary result and the at least one main frame to generate the summary information. Claim 8 A non-temporary recording medium, wherein the non-temporary recording medium is configured to store computer-executable instructions, and the instructions, when executed individually or collectively by one or more processors of an electronic device, cause the electronic device to: display a first image through a display of the electronic device; while displaying the first image through the display, obtain user input requesting the provision of summary information corresponding to a first time interval of the first image from a user; based on the acquisition of the user input, identify context information corresponding to the first time interval of the first image; and using the identified context information, obtain a summary result during the first time interval of the first image through an artificial intelligence learning model of the electronic device, identify at least one major frame among a plurality of frames included within the first time interval of the first image, and provide the summary information including a summary image corresponding to the identified at least one major frame and the obtained summary result. Claim 9 A non-transient recording medium according to claim 8, wherein the instructions further include instructions configured to enable the electronic device to identify the at least one major frame by further utilizing the acquired summary result when executed individually or collectively by the one or more processors. Claim 10 A non-temporary recording medium according to claim 8 or 9, wherein the context information comprises information about a speaker, information about subtitles included in the first image, information about the speaker's utterance, screen information corresponding to a plurality of frames included in the first image, information about the time code of the first image, and / or information about audio included in the first image. Claim 11 A non-transient recording medium characterized in that, in any one of claims 8 to 10, the instructions further comprise instructions configured to cause the electronic device to display the summary information together with the first image through the display when executed individually or collectively by the one or more processors. Claim 12 A non-transient recording medium, wherein, in any one of claims 8 to 11, the instructions further comprise instructions configured to cause the electronic device to identify the at least one major frame among the plurality of frames based on an importance factor (IF) determined based on information regarding the speaker’s emotion identified based on the speaker’s utterance and information regarding the importance of the scene corresponding to the plurality of frames when executed individually or collectively by the one or more processors. Claim 13 A non-transient recording medium characterized in that, in any one of claims 8 to 12, the instructions further comprise instructions configured such that, when executed individually or collectively by the one or more processors, the electronic device: obtains a description of a screen corresponding to a plurality of frames included in the first image using the artificial intelligence learning model, and identifies the obtained description of the screen as the screen information. Claim 14 A non-transient recording medium, characterized in that, in any one of claims 8 to 13, the instructions further comprise, when executed individually or collectively by the one or more processors, an instruction configured to synthesize the summary result and the at least one main frame to generate the summary information. Claim 15 A method for controlling an electronic device, comprising: an operation of displaying a first image through a display of the electronic device; an operation of obtaining a user input requesting the provision of summary information corresponding to a first time interval of the first image from a user while the first image is displayed through the display; an operation of verifying context information corresponding to the first time interval of the first image based on the acquisition of the user input; and using the verified context information: an operation of obtaining a summary result during the first time interval of the first image through an artificial intelligence learning model of the electronic device; an operation of identifying at least one major frame among a plurality of frames included within the first time interval of the first image; and an operation of providing the summary information including a summary image corresponding to the identified at least one major frame and the obtained summary result. Claim 16 A method for controlling an electronic device according to claim 15, further comprising the operation of identifying at least one main frame using the summary result obtained above. Claim 17 A method for controlling an electronic device according to claim 15 or 16, wherein the context information comprises information about a speaker, information about subtitles included in the first image, information about the speaker's utterance, screen information corresponding to a plurality of frames included in the first image, information about the time code of the first image, and / or information about audio included in the first image. Claim 18 A method for controlling an electronic device, characterized in that, in any one of claims 15 to 17, it further includes the operation of displaying the summary information together with the first image through the display. Claim 19 A method for controlling an electronic device, characterized in that, in any one of claims 15 to 18, it further comprises the operation of identifying at least one major frame among the plurality of frames based on an importance factor (IF) determined based on information regarding the speaker’s emotion identified based on the speaker’s utterance and information regarding the importance of the scene corresponding to the plurality of frames. Claim 20 A method for controlling an electronic device, characterized in that, in any one of claims 15 to 19, it further comprises the operation of obtaining a description of a screen corresponding to a plurality of frames included in the first image using the artificial intelligence learning model, and the operation of identifying the obtained description of the screen as the screen information.

Images