Electronic device for generating highlight video based on keywords and operating method thereof

Abstract

An electronic device and a method for generating a highlight video are provided, wherein the electronic device obtains one or more keywords included in a plurality of frames of a video, obtains importance distribution information corresponding to each of the one or more keywords, determines at least one highlight keyword among the one or more keywords based on the importance distribution information, displays the at least one highlight keyword, obtains an input corresponding to at least a portion of the displayed at least one highlight keyword, and generates a highlight video based on a highlight keyword corresponding to the input.

Description

Technical Field

[0001] This disclosure relates to an electronic device and its operating method for generating highlight videos. Specifically, this disclosure relates to an electronic device and its operating method for obtaining keywords from a video and generating the video based on the keywords. Background Technology

[0002] In related technologies, highlight segments are identified based on image analysis (such as frame difference) or speech analysis to generate highlight videos. Accordingly, in these technologies, highlight videos can be generated only when specifically designed for a particular domain (such as sports). Furthermore, even when generating highlight videos, users may find it difficult to interpret the context of the generated highlight videos. For example, users may struggle to identify the criteria by which the highlight video was determined to be a highlight. Additionally, because these technologies process the entire video, they may be difficult to operate in real-time mobile environments.

[0003] There may be a need for a method to generate highlight videos for all types of videos and provide descriptions associated with the highlight videos, so that users can understand and edit the highlight videos.

[0004] The above information is provided for background information purposes only to aid in understanding this disclosure. No decision has been made, nor has any assertion been made, regarding which of the above elements constitutes a prior art application relevant to this disclosure. Summary of the Invention

[0005] Technical solution In embodiments of this disclosure, a method for generating highlight videos based on keywords, executed by an electronic device, is provided. The method may include: obtaining one or more keywords included in multiple frames of a video. The method may include: obtaining importance distribution information corresponding to each of the one or more keywords. The method may include: determining at least one highlight keyword among the one or more keywords based on the importance distribution information. The method may include: displaying at least one highlight keyword. The method may include: obtaining input corresponding to at least a portion of the displayed at least one highlight keyword. The method may include: generating a highlight video based on the highlight keyword corresponding to the input.

[0006] In embodiments of this disclosure, an electronic device for generating highlight videos based on keywords is provided. The electronic device may include at least one processor comprising processing circuitry and a memory storing at least one instruction. When executed individually or jointly by the at least one processor, the at least one instruction causes the electronic device to: obtain one or more keywords included in a plurality of frames of a video. When executed individually or jointly by the at least one processor, the at least one instruction causes the electronic device to: obtain importance distribution information corresponding to each of the one or more keywords. When executed individually or jointly by the at least one processor, the at least one instruction causes the electronic device to: determine at least one highlight keyword among the one or more keywords based on the importance distribution information. When executed individually or jointly by the at least one processor, the at least one instruction causes the electronic device to: display at least one highlight keyword. When executed individually or jointly by the at least one processor, the at least one instruction causes the electronic device to: obtain input corresponding to at least a portion of the displayed at least one highlight keyword. When executed individually or jointly by the at least one processor, the at least one instruction causes the electronic device to: generate a highlight video based on the highlight keyword corresponding to the input.

[0007] In embodiments of this disclosure, a computer-readable recording medium may be provided having a program recorded thereon for performing any of the methods described above and below to operate an electronic device. Attached Figure Description

[0008] Figure 1 This is a diagram illustrating the operation of generating a highlight video based on keywords performed by an electronic device according to an embodiment of the present disclosure.

[0009] Figure 2 This is a flowchart illustrating a method for generating highlight videos based on keywords according to an embodiment of the present disclosure.

[0010] Figure 3 This is an illustration of keywords used to describe a video according to an embodiment of the present disclosure.

[0011] Figure 4 This is a diagram illustrating the importance distribution information corresponding to keywords according to embodiments of the present disclosure.

[0012] Figure 5 This is a diagram used to describe a collection of sections according to embodiments of the present disclosure.

[0013] Figure 6 This is a diagram used to describe a collection of sections according to embodiments of the present disclosure.

[0014] Figure 7This is a diagram illustrating the process of generating a highlight video according to an embodiment of the present disclosure.

[0015] Figure 8a This is a diagram illustrating a module included in an electronic device 100 according to an embodiment of the present disclosure.

[0016] Figure 8b This is a diagram illustrating a module included in an electronic device 100 according to an embodiment of the present disclosure.

[0017] Figure 9 This is a diagram illustrating multiple groups obtained by dividing a video according to an embodiment of the present disclosure.

[0018] Figure 10 This is a diagram illustrating the process of generating a highlight video according to embodiments of the present disclosure.

[0019] Figure 11 This is a diagram illustrating the process of determining collection keywords and general keywords according to embodiments of the present disclosure.

[0020] Figure 12 This is a diagram illustrating a user interface (UI) of an electronic device 100 displaying a collection of keywords according to an embodiment of the present disclosure.

[0021] Figure 13 This is a diagram illustrating a UI for editing video using an electronic device 100 according to an embodiment of the present disclosure.

[0022] Figure 14 This is an illustration of a UI showing search results including images and / or videos according to embodiments of this disclosure.

[0023] Figure 15 This is a diagram illustrating a UI for editing multiple videos by an electronic device 100 according to an embodiment of the present disclosure.

[0024] Figure 16 This is an illustration of a UI for sharing highlight videos by an electronic device 100 based on keywords, according to an embodiment of the present disclosure.

[0025] Figure 17 This is an illustration of a UI showing search results including images and / or videos according to embodiments of this disclosure.

[0026] Figure 18 This is a block diagram illustrating the configuration of an electronic device according to an embodiment of the present disclosure.

[0027] Figure 19 This is a block diagram illustrating the configuration of an electronic device according to an embodiment of the present disclosure. Detailed Implementation

[0028] In embodiments of this disclosure, the expression "at least one of a, b, or c" may refer to "only a", "only b", "only c", "both a and b", "both a and c", "both b and c", "all of a, b, and c", or variations thereof.

[0029] In embodiments of this disclosure, the expression "a, b, and / or c" may be interpreted as the same as the expression "at least one of a, b, or c". For example, the expression "a, b, and / or c" may refer to "only a", "only b", "only c", "both a and b", "both a and c", "both b and c", "all of a, b, and c", or variations thereof.

[0030] Although the terms used in the embodiments of this disclosure are selected from commonly used terms that are currently widely used in consideration of their function in the embodiments of this disclosure, these terms may vary depending on the intent of those skilled in the art, precedent, or the emergence of new technologies. Furthermore, in certain cases, the terms are arbitrarily chosen by the applicant, and in such cases, their meaning can be understood through the corresponding description. Therefore, the terms used in the embodiments of this disclosure are not merely names of terms, but are defined based on the meaning of the terms and the entire disclosure.

[0031] In embodiments of this disclosure, singular expressions may include plural expressions unless the context explicitly indicates otherwise. For example, the expression "constituting a surface" may also include referring to one or more such surfaces. It will be understood that although the terms "first," "second," etc., may be used to describe various elements in embodiments of this disclosure, these elements should not be limited by these terms. These terms are used only to distinguish one component from another.

[0032] In embodiments of this disclosure, when a component "comprises" or "includes" an element, it should be understood that the component may also include other elements, but does not exclude other elements, unless there is a specific description to the contrary. In embodiments of this disclosure, terms such as "...unit," "module," etc., indicate a unit that processes at least one function or operation, and that unit may be implemented in hardware or software, or in a combination of hardware and software.

[0033] In embodiments of this disclosure, the expression "configured as (or set to)" may be replaced, as appropriate, with terms such as "suitable for," "capable of," "designed for," "suitable for," "manufactured as," or "capable of." The expression "configured as (or set to)" may not necessarily mean "specifically designed for" at the hardware level. Optionally, in some cases, the expression "system configured as..." may indicate that the system, together with other devices or components, is "capable of...". For example, "processor configured as (or set to) perform A, B, and C" may refer to a dedicated processor (e.g., an embedded processor) for performing the respective operations, or a general-purpose processor (e.g., a central processing unit (CPU) or application processor (AP)) capable of performing the respective operations by running one or more software programs stored in memory.

[0034] In embodiments of this disclosure, it should be understood that when components are “connected” or “combined” with each other, unless otherwise stated, components may be directly connected or combined with each other, but may optionally be connected or combined with each other when there are elements therebetween.

[0035] When describing this disclosure herein, descriptions of techniques well-known in the art to which this disclosure pertains and not directly related to this disclosure may be omitted. By omitting unnecessary explanations, the essence of this disclosure remains unambiguous and is clearly conveyed. In the accompanying drawings, for the purpose of more clearly describing this disclosure, parts or elements unrelated to this disclosure are omitted, and throughout the specification, the same reference numerals denote the same components. The dimensions of each component do not exactly correspond to the actual dimensions of each component. In the accompanying drawings, the same or corresponding components are indicated by the same reference numerals.

[0036] The advantages and features of this disclosure, as well as methods of implementing them, will become apparent from the embodiments described in detail below with reference to the accompanying drawings. However, this disclosure may be implemented in many different forms and should not be construed as limited to these embodiments. The disclosed embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of this disclosure to those skilled in the art only by means of the claims. Embodiments of this disclosure may be defined by the claims.

[0037] In embodiments of this disclosure, each block of a flowchart illustration and combinations of blocks in a flowchart illustration may be implemented by computer program instructions. The computer program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus, and the instructions, which are executable by the processor of the computer or other programmable data processing apparatus, may generate means for performing the functions specified in the flowchart blocks. The computer program instructions may also be stored in a computer-executable memory or a computer-readable memory that can direct a computer or other programmable data processing apparatus to perform its functions in a particular manner, and the instructions stored in the computer-executable memory or computer-readable memory produce an article of writing including instruction means for performing the functions specified in the flowchart blocks. The computer program instructions may also be loaded onto a computer or other programmable data processing apparatus. It should be understood that each block in a flowchart and combination of flowcharts may be executed by one or more computer programs that include computer-executable instructions. One or more computer programs may be stored entirely in a single memory or may be stored in a distributed manner across multiple different memories. In embodiments, the memory may include one or more storage media that store instructions.

[0038] All the functions or operations described in the embodiments of this disclosure may be processed by a single processor or a combination of processors. A single processor or combination of processors may correspond to circuitry configured to perform processing and may include circuitry such as an application processor (AP), a communication processor (CP), a graphics processing unit (GPU), a neural processing unit (NPU), a microprocessor unit (MPU), a system-on-a-chip (SoC), or an integrated circuit (IC).

[0039] At least one processor according to embodiments of this disclosure may include various types of processing circuitry and / or multiple processors. For example, the term "processor" as used in embodiments of this disclosure (including in the claims) may include various types of processing circuitry comprising at least one processor, wherein one or more of the at least one processor are configured to individually and / or collaboratively perform the various functions described herein in a distributed manner. As used herein, when "processor," "at least one processor," and "one or more processors" are described as being configured to perform various functions, such terms may include, for example, but not limited to, cases where a single processor performs some of the described functions and other processors perform other functions of the described functions, and there are cases where a single processor can perform all of the described functions. Furthermore, at least one processor may include a combination of processors performing various listed / disclosed functions in a distributed manner. At least one processor may execute program instructions to implement or perform various functions.

[0040] In embodiments of this disclosure, each block in the flowchart may represent a portion of a module, segment, or code that includes one or more executable instructions for performing a specified logical function. In embodiments of this disclosure, the functions mentioned in the blocks may also occur out of order. For example, two blocks shown consecutively may be executed substantially simultaneously, or the blocks may be executed in reverse order depending on the functions involved.

[0041] The term "...unit" as used in embodiments of this disclosure may refer to a software or hardware component, such as a field-programmable gate array (FPGA) or application-specific integrated circuit (ASIC), and the "...unit" performs a specific function. The "...unit" is not limited to software or hardware. The "...unit" may be configured to reside in addressable memory or may be configured to operate one or more processors. In embodiments of this disclosure, the "...unit" may include components (such as software components, object-oriented software components, class components, and task components), processes, functions, attributes, procedures, subroutines, program code segments, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. Functionality provided by a particular component or a particular "...unit" may be combined to reduce its number or divided into additional components. In embodiments of this disclosure, the "...unit" may include one or more processors.

[0042] The artificial intelligence-related functions according to embodiments of this disclosure are operated via a processor and memory. The processor may consist of one or more processors. In this case, the one or more processors may each be a general-purpose processor (such as a CPU, AP, digital signal processor (DSP), etc.), a graphics-specific processor (such as a GPU, visual processing unit (VPU)), or an artificial intelligence-specific processor (such as an NPU). The one or more processors perform control to process input data according to predefined operating rules or artificial intelligence models stored in memory. Optionally, when the one or more processors are each artificial intelligence-specific processors, the artificial intelligence-specific processors may be designed with hardware architectures dedicated to processing specific artificial intelligence models.

[0043] Predefined operating rules or artificial intelligence models can be generated via a training process. Here, "generated via a training process" can mean generating predefined operating rules or artificial intelligence models configured to perform desired characteristics (or purposes) by training a basic artificial intelligence model using a learning algorithm that utilizes a large amount of training data. Such training can be performed by an apparatus implementing artificial intelligence according to embodiments of this disclosure, or by a separate server and / or system. Examples of learning algorithms may include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0044] Artificial intelligence models may include multiple neural network layers. Each of these layers has multiple weight values ​​and performs neural network operations by operating on the results of the previous layer's operations and these weight values. The weight values ​​of the multiple neural network layers can be optimized based on the training results of the artificial intelligence model. For example, the weight values ​​can be updated to reduce or minimize the loss or cost values ​​acquired by the artificial intelligence model during training. Examples of artificial neural networks may include, but are not limited to, deep neural networks (DNNs), such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), restricted Boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), or deep Q-networks.

[0045] According to embodiments of this disclosure, a method for generating a highlight video by an electronic device may include: using frames of the video as input data for an artificial intelligence model, and obtaining output data by identifying keywords corresponding to at least a portion of the frames, in order to identify the keywords. The artificial intelligence model may be generated via a training process. Here, generating via a training process may mean generating a predefined operating rule or artificial intelligence model configured to perform a desired characteristic (or purpose) by training a basic artificial intelligence model using a learning algorithm utilizing a large amount of training data. The artificial intelligence model may include multiple neural network layers. Each neural network layer may include multiple weight values, and neural network arithmetic operations may be performed via arithmetic operations between the result of arithmetic operations of the previous layer and the multiple weight values.

[0046] Visual understanding is a technology that identifies and processes objects in the same way as human vision, and includes object recognition, object tracking, image retrieval, human recognition, scene recognition, 3D reconstruction / localization, image enhancement, and more.

[0047] In the following, embodiments of the present disclosure will be described more fully with reference to the accompanying drawings to enable those skilled in the art to perform the embodiments without difficulty. However, the present disclosure may be implemented in many different forms and should not be construed as limited to the embodiments set forth herein. In the drawings, portions or elements unrelated to the present disclosure have been omitted for clarity, and throughout the specification, the same reference numerals denote the same components. Furthermore, reference numerals used in each drawing are used only to describe that drawing, and different reference numerals used in each drawing in different drawings are not used to indicate different components. The present disclosure will now be described in detail with reference to the accompanying drawings.

[0048] In this disclosure, "video" may include a plurality of still images. A video may include a plurality of still images in a time sequence. For example, a video can represent movement of an object included in the images by configuring images captured sequentially in a time sequence. In embodiments of this disclosure, still images included in a video may be referred to as frames. In embodiments of this disclosure, a video may also include audio data (such as speech or music).

[0049] In this disclosure, "keywords" may include words or phrases that indicate features associated with a video or a frame of a video. For example, keywords may include words or phrases that include information about at least one of an emotion, object, action, behavior, situation, location, or event associated with a video or a frame of a video. For example, keywords may indicate at least one of an emotion, object, action, behavior, situation, location, or event included in a video or a frame of a video. Keywords may be stored in memory as information associated with a video or a frame of a video.

[0050] In this disclosure, "importance distribution information" may include the distribution of the importance of keywords in a video. Importance scores may indicate the importance of keywords in a video. In embodiments of this disclosure, an importance score may be determined to be high when a keyword is frequently identified in a portion of the video. An importance score may be determined to be low when a keyword is identified multiple times throughout the entire video. For example, the importance score for the keyword "pet" may be determined to be lower when a pet is identified for 50 seconds in a 1-minute video than when a pet is identified for 10 seconds. Importance distribution information may include time-series data. Importance distribution information may include importance scores corresponding to time. For example, importance distribution information may include importance scores based on the playback time of the video.

[0051] In this disclosure, "highlight keywords" may include keywords identified in small segments of a video. Highlight keywords can be determined based on importance distribution information. Keywords whose importance scores in the importance distribution information meet defined conditions can be identified as highlight keywords. For example, keywords whose maximum or average importance scores are greater than a threshold can be identified as highlight keywords.

[0052] In this disclosure, "highlight video" can refer to an image generated using a portion of an original video. For example, a highlight video may include frames corresponding to highlight segments across multiple frames of the original video. In embodiments of this disclosure, highlight segments may be determined based on the importance score of highlight keywords. For example, a defined segment around the highest point of a highlight keyword may be determined as a highlight segment.

[0053] Figure 1 This is a diagram illustrating the operation of generating a highlight video based on keywords performed by an electronic device according to an embodiment of the present disclosure.

[0054] Reference Figure 1 The electronic device 100 may be a mobile device, such as a smartphone, tablet PC, laptop computer, digital camera, e-book terminal, digital broadcasting terminal, personal digital assistant (PDA), portable multimedia player (PMP), navigation device, or MP3 player. In embodiments of this disclosure, the electronic device 100 may be a household appliance, such as a TV, air conditioner, robotic cleaner, or garment care machine. However, this disclosure is not limited thereto, and in another embodiment of this disclosure, the electronic device 100 may be implemented as a wearable device, such as a smartwatch, an augmented reality glasses device (e.g., augmented reality (AR) glasses), a head-mounted device (HMD), or a body-attachable device (e.g., a skin patch).

[0055] Reference Figure 1 The electronic device 100 can obtain keywords 110 for the video. In embodiments of this disclosure, the electronic device 100 can obtain one or more keywords included in multiple frames of the video. The electronic device 100 can obtain keywords corresponding to frames by analyzing the frames of the video. For example, the electronic device 100 can obtain keywords indicating emotions, objects, actions, behaviors, locations, situations, or events included in the frames of the video. For example, keywords may include kissing, beach, pet, hugging, woman, smiling, and love. The electronic device 100 can obtain keywords 110 for the video based on keywords for each frame of the video.

[0056] The electronic device 100 can obtain importance distribution information 120 corresponding to the keyword 110. In embodiments of this disclosure, the electronic device 100 can obtain importance distribution information 120 corresponding to each of one or more keywords. For example, the electronic device 100 can obtain importance distribution information 120 corresponding to "hug" and importance distribution information 120 corresponding to "smile".

[0057] In embodiments of this disclosure, the importance distribution information 120 may include a time-corresponding importance score. When a keyword is identified in a portion of the video, the importance score may be determined to be high. For example, when the video includes frames related to "hugging" in a time period from 130 seconds to 310 seconds, the importance score for "hugging" may be determined to be high in that time period. When a keyword is identified throughout the entire video, the importance score may be determined to be low. For example, when the video includes frames related to "pets" throughout the entire video, the importance score for "pets" may be determined to be low throughout the entire video.

[0058] Electronic device 100 can determine highlight keywords 130. In embodiments of this disclosure, electronic device 100 can determine highlight keywords 130 based on importance distribution information 120. When the importance score of importance distribution information 120 meets the defined conditions, electronic device 100 can determine the keyword corresponding to importance distribution information 120 as a highlight keyword. For example, electronic device 100 can determine "hug" as a highlight keyword based on the maximum value of its importance score being greater than a threshold. For example, "pet" may not be determined as a highlight keyword if the maximum value of its importance score is less than a threshold.

[0059] In embodiments of this disclosure, electronic device 100 may identify at least some of the multiple keywords of a video as highlight keywords based on importance distribution information 120.

[0060] Electronic device 100 can generate highlight video 150 based on highlight keywords 130. Electronic device 100 can generate highlight video 150 to include frames corresponding to highlight keywords 130.

[0061] In embodiments of this disclosure, electronic device 100 may display highlight keywords 130. Electronic device 100 may also display a user interface (UI) 140 corresponding to highlight keywords 130. For example, electronic device 100 may display UI 140 corresponding to "kiss," "hug," and "smile." Electronic device 100 may display highlight keywords 130 without displaying a portion of it, based on the size of UI 140 and the size of the display area.

[0062] Electronic device 100 can receive input corresponding to UI 140. For example, electronic device 100 can receive touch input to an area displaying UI 140. For example, electronic device 100 can receive voice input corresponding to highlight keyword 130. In embodiments of this disclosure, electronic device 100 can generate highlight video 150 based on highlight keyword 130 corresponding to the input. Electronic device 100 can determine highlight segments based on highlight keyword 130 corresponding to the input. Highlight segments can be determined based on importance score of highlight keyword 130. For example, highlight segments can be determined as segments with importance scores greater than a threshold. For example, a highlight segment for "hug" can be determined to be between 130 seconds and 310 seconds. Electronic device 100 can generate highlight video to include frames included in the highlight segments.

[0063] Figure 2 This is a flowchart illustrating a method for generating highlight videos based on keywords according to an embodiment of the present disclosure.

[0064] In embodiments of this disclosure, the method for generating highlight videos can be performed by electronic device 100. For example, electronic device 100 can perform each operation of the image decoding method by executing at least one instruction included in memory by its processor.

[0065] In operation S210, the method may include obtaining one or more keywords included in multiple frames of a video. Electronic device 100 may obtain one or more keywords included in multiple frames of a video. For example, electronic device 100 may extract keywords for each frame of the video. Keywords may include words indicating at least a portion of a frame. In embodiments of this disclosure, electronic device 100 may select keywords for a frame from words stored in a database (DB).

[0066] In embodiments of this disclosure, electronic device 100 can obtain keywords included in a frame by using an artificial intelligence model. The artificial intelligence model can be trained to output keywords included in the frame. The number of keywords output by the artificial intelligence model is unlimited, and the artificial intelligence model can output one or more keywords. When there are no features in the frame, the artificial intelligence model may not output keywords. Electronic device 100 can obtain keywords and / or confidence scores corresponding to keywords by inputting frames of video into the artificial intelligence model. The confidence score can refer to the probability that the artificial intelligence model determines the output keyword is included in the frame.

[0067] In embodiments of this disclosure, the electronic device 100 can obtain keywords for each frame while capturing video. For example, the electronic device 100 can obtain keywords for each frame captured in real time while capturing video. The electronic device 100 can extract keywords from frames independently of video capture.

[0068] In embodiments of this disclosure, the electronic device 100 may obtain keywords for each frame after video capture is complete. For example, the electronic device 100 may obtain keywords for each frame during post-processing after video capture is complete.

[0069] In operation S220, the method may include obtaining importance distribution information corresponding to each of the one or more keywords. The electronic device 100 may obtain importance distribution information corresponding to each of the one or more keywords.

[0070] In embodiments of this disclosure, importance distribution information may include an importance score corresponding to time. When a keyword is identified only in a portion of the video, the importance score may be determined as high. When a keyword is identified throughout the entire video or the frequency of keyword identification is low, the importance score may be determined as low.

[0071] In operation S230, the method may include determining at least one summary keyword from one or more keywords based on importance distribution information. The electronic device 100 may determine at least one summary keyword from one or more keywords based on importance distribution information.

[0072] In embodiments of this disclosure, when the importance score of the importance distribution information meets the defined conditions, the electronic device 100 can determine the keywords corresponding to the importance distribution information as collection keywords.

[0073] In operation S240, the method may include displaying at least one highlight keyword. The electronic device 100 may display at least one highlight keyword. In embodiments of this disclosure, the electronic device 100 may display a UI corresponding to the highlight keyword via a display.

[0074] In operation S250, the method may include obtaining input corresponding to at least a portion of at least one of the displayed highlight keywords. The electronic device 100 may obtain input corresponding to at least a portion of the displayed highlight keywords. In embodiments of this disclosure, the electronic device 100 may obtain touch input or voice input corresponding to the highlight keywords. For example, the electronic device 100 may obtain touch input to an area of ​​the UI displaying the highlight keyword, or it may obtain voice input indicating the highlight keyword.

[0075] In operation S260, the method may include generating a highlight video based on highlight keywords corresponding to the input. The electronic device 100 may generate a highlight video based on highlight keywords corresponding to the input.

[0076] In embodiments of this disclosure, electronic device 100 may determine highlight segments corresponding to input highlight keywords. Electronic device 100 may generate highlight videos based on frames included in the highlight segments.

[0077] In embodiments of this disclosure, electronic device 100 can generate highlight videos that reflect the user's intent by generating highlight videos corresponding to the input. Electronic device 100 can extract keywords from the video and generate highlight videos based on those keywords, thereby providing the user with a reason for generating the highlight videos, and the user can modify the highlight videos by using the keywords. For example, a user can request highlight videos from electronic device 100 based on keywords of interest.

[0078] Figure 3 This is an illustration of keywords used to describe a video according to an embodiment of the present disclosure.

[0079] In embodiments of this disclosure, electronic device 100 can determine keywords based on video. Electronic device 100 can obtain keywords corresponding to frames of the video.

[0080] Reference Figure 3 For example, a frame of the video could be a scene of a girl and a boy feeding animals outdoors. In embodiments of this disclosure, keywords corresponding to the frame could include "child," "girl," "joy," "park," "summer," "people," "playing," and "cute."

[0081] In embodiments of this disclosure, multiple defined keywords may be related. For example, in response to a frame including a girl, "child," "girl," and "person" may be selected. Electronic device 100 may determine multiple keywords corresponding to an object included in a frame. However, this disclosure is not limited thereto, and the number of keywords corresponding to an object may be limited. For example, the number of keywords corresponding to an object may be limited to one.

[0082] In embodiments of this disclosure, keywords may include feature information about the video expressed in natural language. In embodiments of this disclosure, electronic device 100 may obtain feature information indicating at least one of emotion, object, action, behavior, situation, location, or event included in frames of a video. Electronic device 100 may determine keywords for the video to include keywords corresponding to the feature information. For example, electronic device 100 may determine keywords for the video to include a first keyword included in a first frame and a second keyword included in a second frame.

[0083] In embodiments of this disclosure, electronic device 100 may determine keywords using an artificial intelligence (AI) model. The AI ​​model may be an AI model trained to receive video frames as input and output keywords corresponding to at least one of the following: emotion, object, action, behavior, situation, location, or event included in the frame. When no keyword corresponds to a frame, electronic device 100 may be uncertain about the keyword or may determine that no keyword exists. In embodiments of this disclosure, the AI ​​model may output keywords and a confidence score for the corresponding keyword. The confidence score may indicate the probability that the AI ​​model determines the output keyword is included in the frame. For example, a keyword with a high confidence score may be determined by the AI ​​model to have a high probability of being included in the frame.

[0084] In embodiments of this disclosure, the electronic device 100 may select a keyword corresponding to at least a portion of a frame from a plurality of defined keywords. For example, the electronic device 100 may select a keyword indicated by at least a portion of a frame from a plurality of keywords included in a keyword DB.

[0085] Figure 4 This is a diagram illustrating the importance distribution information corresponding to keywords according to embodiments of the present disclosure.

[0086] Reference Figure 4This shows the importance distribution information for each keyword. For example, Figure 4 The importance distribution information shown may correspond to each keyword included in the video. In embodiments of this disclosure, the importance distribution information may include the distribution of the importance of keywords in the video.

[0087] In embodiments of this disclosure, importance distribution information may include information about the importance scores of frames based on time or video. Frame numbers may refer to the order in which frames are displayed. For example, frame numbers may be displayed in ascending order, starting from 0. Importance distribution information may be represented as information about importance scores based on time.

[0088] In embodiments of this disclosure, an importance score can indicate the importance of a keyword in a video. The importance score can be determined based on the frequency of the keyword. For example, it can be determined based on the number of times a keyword is identified in a portion of the video.

[0089] In embodiments of this disclosure, an importance score can be determined to be high when a keyword is frequently identified in a segment of the video. A first distribution 410 indicates importance distribution information corresponding to the keyword "kiss." The first distribution 410 has a high importance score between 150 seconds and 250 seconds. The first distribution 410 may indicate that the keyword "kiss" is identified multiple times in the segment of the video between 150 seconds and 250 seconds.

[0090] In embodiments of this disclosure, when a keyword is identified multiple times throughout a segment of the video, its importance score can be determined to be low. For example, a keyword identified multiple times throughout the segment can be determined to have low importance for generating a highlight video. A second distribution 420 indicates the importance distribution information corresponding to the keyword "walking". The second distribution 420 has a low importance score in each frame of the video. The second distribution 420 may indicate that the keyword "walking" is identified multiple times throughout the segment of the video.

[0091] In embodiments of this disclosure, an importance score can be determined to be low when a keyword is rarely identified throughout the entire segment of a video. For example, a keyword that is rarely identified throughout the entire segment can be determined to have low importance for generating a highlight video. The second distribution 420 may indicate that the keyword "walking" is almost never identified throughout the entire segment of the video.

[0092] In embodiments of this disclosure, the electronic device 100 can obtain keywords included in frames of a video and can obtain importance distribution information corresponding to the keywords. The electronic device 100 can determine highlight keywords based on the importance distribution information. In embodiments of this disclosure, the electronic device 100 can determine highlight keywords of a video from keywords included in frames of a video. In embodiments of this disclosure, highlight keywords may refer to keywords that indicate the moment of the highlight while having relatively high rarity.

[0093] In embodiments of this disclosure, the electronic device 100 can determine highlight keywords based on importance scores. In embodiments of this disclosure, the electronic device 100 can identify keywords whose importance scores meet defined conditions as highlight keywords. For example, the electronic device 100 can identify keywords whose maximum importance score is greater than a threshold as highlight keywords. For example, the electronic device 100 can identify keywords whose average importance score is greater than a threshold as highlight keywords. In embodiments of this disclosure, the electronic device 100 can identify only a defined number of keywords as highlight keywords in descending order of importance score. In embodiments of this disclosure, the electronic device 100 can determine that keywords whose importance scores do not meet defined conditions are not highlight keywords.

[0094] Figure 5 This is a diagram used to describe a collection of sections according to embodiments of the present disclosure.

[0095] In embodiments of this disclosure, electronic device 100 may determine highlight segments based on importance distribution information. The importance distribution information may include importance scores for keywords based on video frames or time. See also... Figure 5 This illustrates the importance distribution information of exemplary keywords. Frame F1 can refer to the frame with the highest importance score.

[0096] In embodiments of this disclosure, electronic device 100 may determine highlight segments based on importance distribution information. Electronic device 100 may determine highlight segments based on importance scores. Electronic device 100 may determine highlight segments based on the maximum value of the importance score. Electronic device 100 may determine a defined segment including the maximum value of the importance score as a highlight segment. For example, electronic device 100 may determine a defined segment including frame F1 as a highlight segment. Electronic device 100 may determine a highlight segment to include frames with the maximum value of the importance score and frames with importance scores greater than a threshold. For example, electronic device 100 may determine a highlight segment to include frame F1 and frames with an importance score of 40% or more of the maximum value.

[0097] In embodiments of this disclosure, electronic device 100 may determine the start and end frames of a highlight segment. For example, the importance score of frame F2 may be 40% of the importance score of frame F1, and frame F3 may be the last frame of the video. Electronic device 100 may store information about the start and end frames of the highlight segment.

[0098] In embodiments of this disclosure, electronic device 100 may determine multiple highlight segments based on importance distribution information. For example, multiple highlight segments may be identified as local maxima that include importance distribution information. For instance, the importance distribution information has local maxima in frames F1 and F4. Therefore, in this example, electronic device 100 may determine a first highlight segment including F1 and a second highlight segment including F4. In embodiments of this disclosure, electronic device 100 may determine multiple highlight segments by using an artificial intelligence model. For example, electronic device 100 may determine multiple highlight segments by using a Gaussian mixture model (GMM).

[0099] In embodiments of this disclosure, the electronic device 100 may determine the highlight segment based solely on highlight keywords. The electronic device 100 may determine highlight keywords based on importance distribution information and may determine the highlight segment corresponding to the highlight keywords.

[0100] Figure 6 This is a diagram used to describe a collection of sections according to embodiments of the present disclosure.

[0101] In embodiments of this disclosure, electronic device 100 may determine highlight segments based on importance distribution information. The importance distribution information may include importance scores for keywords based on video frames or time. See also... Figure 6 This shows the importance distribution information of exemplary keywords.

[0102] In embodiments of this disclosure, electronic device 100 may determine highlight segments based on importance distribution information. Electronic device 100 may determine highlight segments based on importance scores. Electronic device 100 may determine highlight segments based on importance scores. Electronic device 100 may define a segment comprising frames having an importance score greater than a threshold as a highlight segment.

[0103] In embodiments of this disclosure, electronic device 100 may determine the start and end frames of a highlight segment. The start and end frames of the highlight segment may each be one of the first frame of the video, the last frame of the video, or a frame with an importance score equal to a threshold. For example, the start frame F1 of the highlight segment may have an importance score equal to the threshold, and the end frame F2 of the highlight segment may be the last frame of the video. Electronic device 100 may store information about the start and end frames of the highlight segment.

[0104] In embodiments of this disclosure, electronic device 100 may determine multiple highlight segments based on importance distribution information. For example, electronic device 100 may determine multiple highlight segments with importance scores greater than a threshold.

[0105] In embodiments of this disclosure, the electronic device 100 may determine the highlight segment based solely on highlight keywords. The electronic device 100 may determine highlight keywords based on importance distribution information and may determine the highlight segment corresponding to the highlight keywords.

[0106] Figure 7 This is a diagram illustrating the process of generating a highlight video according to an embodiment of the present disclosure.

[0107] In embodiments of this disclosure, electronic device 100 can generate a highlight video 730 corresponding to highlight keyword 710. Electronic device 100 can display highlight keyword 710. For example, electronic device 100 can output a UI corresponding to highlight keyword 710 using a display.

[0108] In embodiments of this disclosure, electronic device 100 may receive input corresponding to highlight keywords 710. A user may select at least a portion of the highlight keywords 710 displayed on a display. Electronic device 100 may receive input for selecting at least a portion of the highlight keywords 710. For example, a user may select "kiss" from the highlight keywords 710 displayed on the display, and electronic device 100 may receive input for selecting "kiss" from the highlight keywords 710. The input may include at least one of voice input or touch input.

[0109] In embodiments of this disclosure, electronic device 100 can determine a highlight segment based on highlight keywords 710 corresponding to the input. Electronic device 100 can identify highlight keywords 710 corresponding to the input and can determine the highlight segment based on importance distribution information 720 of the highlight keywords 710. For example, electronic device 100 can identify "kiss" corresponding to the input from the highlight keywords 710 and can determine the highlight segment based on the importance distribution information 720 of "kiss".

[0110] In embodiments of this disclosure, electronic device 100 may determine a highlight segment based on a plurality of highlight keywords 710 corresponding to the input.

[0111] Electronic device 100 can receive input for multiple highlight keywords. In embodiments of this disclosure, electronic device 100 can receive multiple inputs indicating highlight keywords 710. For example, electronic device 100 can receive a first input indicating the first highlight keyword "kiss" and a second input indicating the second highlight keyword "love". In embodiments of this disclosure, electronic device 100 can receive input indicating multiple highlight keywords 710. For example, electronic device 100 can receive one input indicating the first highlight keyword "kiss" and the second highlight keyword "love".

[0112] Electronic device 100 can determine multiple highlight segments based on importance distribution information 720 corresponding to each highlight keyword 710. For example, electronic device 100 can determine a first highlight segment based on first importance distribution information corresponding to the first highlight keyword "kiss," and can determine a second highlight segment based on second importance distribution information corresponding to the second highlight keyword "love." In this example, electronic device 100 can determine highlight segments to include both the first and second highlight segments.

[0113] In embodiments of this disclosure, electronic device 100 may generate a highlight video 730 corresponding to a highlight segment. Electronic device 100 may generate highlight video 730 to include multiple frames of video corresponding to the highlight segment.

[0114] Figure 8a This is a diagram illustrating a module included in an electronic device 100 according to an embodiment of the present disclosure.

[0115] In embodiments of this disclosure, the electronic device 100 may include a keyword extraction module 810, a segment sampling module 820, an importance distribution determination module 830, a highlight keyword determination module 840, and a highlight video generation module 850.

[0116] Included Figure 8a The keyword extraction module 810, segment sampling module 820, importance distribution determination module 830, highlight keyword determination module 840, and highlight video generation module 850 in the electronic device 100 can be components based on function or role classification. Figure 8aThe keyword extraction module 810, segment sampling module 820, importance distribution determination module 830, highlight keyword determination module 840, and highlight video generation module 850 in the electronic device 100 can be software configurations implemented by the processor of the electronic device 1000 by executing programs or instructions stored in memory, and can also be virtual configurations with actual matching hardware devices. In other words, the operations performed by the processor of the electronic device 1000 by executing programs or instructions stored in memory can be classified into multiple groups for each function or purpose, and the entities performing the various operations included in the classified groups can be represented as including Figure 8a The electronic device 100 includes a keyword extraction module 810, a segment sampling module 820, an importance distribution determination module 830, a highlight keyword determination module 840, and a highlight video generation module 850. These modules can be executed by the electronic device 100 or its processor by executing programs or instructions stored in memory. Figure 8a The operations and functions performed by the keyword extraction module 810, the segment sampling module 820, the importance distribution determination module 830, the summary keyword determination module 840, and the summary video generation module 850.

[0117] In embodiments of this disclosure, the keyword extraction module 810 can obtain keywords based on video. The keyword extraction module 810 can obtain keywords corresponding to each frame of the video. The keyword extraction module 810 may include an artificial intelligence model that outputs keywords by using frames as input. Keywords may indicate at least a partial region of a frame in the video.

[0118] In embodiments of this disclosure, the segment sampling module 820 can divide a video into multiple groups. The segment sampling module 820 can divide the video into multiple groups based on a division interval. For example, the segment sampling module 820 can determine the groups of the video in units of 30 frames. The division interval can represent a value determined to equally divide the video. In embodiments of this disclosure, reference will be made to... Figure 9 Describe in detail the process of dividing the video into multiple groups.

[0119] In embodiments of this disclosure, the keyword extraction module 810 and the segment sampling module 820 are connected sequentially, but can be processed in parallel, and the order between the keyword extraction module 810 and the segment sampling module 820 can be interchanged.

[0120] In embodiments of this disclosure, the importance distribution determination module 830 can obtain importance distribution information corresponding to keywords. In embodiments of this disclosure, the importance distribution determination module 830 can obtain importance distribution information including importance scores based on time or frame. In embodiments of this disclosure, the importance distribution determination module 830 can determine importance scores based on the frequency of keywords included in multiple groups within a video.

[0121] In embodiments of this disclosure, the importance distribution determination module 830 can identify the number of frames including the first keyword in a first group of multiple groups. For example, when the division interval is 30, the importance distribution determination module 830 can identify how many frames in 30 frames include the first keyword. The importance distribution determination module 830 can identify the number of groups including the first keyword in multiple groups. For example, when the division interval is 30 and the entire video corresponds to 300 frames, the importance distribution determination module 830 can identify the number of groups including the first keyword in 10 groups. The importance distribution determination module 830 can obtain importance distribution information corresponding to the first keyword based on the number of frames including the first keyword and the number of groups including the first keyword. For example, the importance distribution determination module 830 can determine a higher importance score as the number of frames including the first keyword increases, and a lower importance score as the number of groups including the first keyword increases. In embodiments of this disclosure, the importance score can be determined based on Equation 1.

[0122] [Equation 1]

[0123] Here, k represents the keyword index, d represents the group index, and TF... k,d This represents the number of keywords included in the group, N represents the total number of groups in the video, and df k This indicates the number of groups that include the keyword.

[0124] In embodiments of this disclosure, the importance distribution determination module 830 may obtain importance distribution information based on the ratio between the number of groups including the first keyword and the number of all groups included in the video. The importance distribution determination module 830 may determine the ratio between the number of groups including the first keyword and the number of all groups included in the video. For example, the importance distribution determination module 830 may determine N / df in Equation 1. k And it can be based on N / df k Obtain importance distribution information.

[0125] In embodiments of this disclosure, the importance distribution determination module 830 can obtain importance distribution information based on the ratio between the number of frames in a first group including the first keyword and the total number of keywords included in the first group. The importance distribution determination module 830 can determine the ratio between the number of frames in a first group including the first keyword and the total number of keywords included in the first group. For example, the importance distribution determination module 830 can determine the TF in Equation 1. k,d The ratio between the number of keywords and the number of all keywords included in the first group d, and importance distribution information can be obtained based on the determined ratio.

[0126] In embodiments of this disclosure, importance scores may be determined based on Equation 2 or Equation 3.

[0127] [Equation 2]

[0128] Here, c k,d This represents the confidence score for the keyword, and the other variables may have the same meaning as in Equation 1.

[0129] In embodiments of this disclosure, importance scores may be determined based on Equation 3.

[0130] [Equation 3]

[0131] In embodiments of this disclosure, the importance distribution determination module 830 is not limited to Equations 1 to 3, and can determine an importance score such that the importance score increases when the number of keywords included in a group increases, and decreases when the number of groups including keywords increases.

[0132] In embodiments of this disclosure, the importance distribution determination module 830 may obtain a confidence score indicating whether each frame included in the first group contains a first keyword. For example, the importance distribution determination module 830 may obtain c from Equation 2. k,d In embodiments of this disclosure, a confidence score can be determined from the artificial intelligence model of the keyword extraction module 810. For example, the artificial intelligence model of the keyword extraction module 810 can determine a confidence score corresponding to the first keyword for each frame, and c in Equation 2... k,d It can refer to the average confidence score of the first keyword included in the first group. According to an embodiment of this disclosure, the importance distribution determination module 830 can obtain importance distribution information corresponding to the first keyword based on multiple confidence scores, the number of frames including the first keyword, and the number of groups including the first keyword.

[0133] In embodiments of this disclosure, the importance distribution determination module 830 can obtain importance distribution information based on the importance score for each segment. The importance distribution determination module 830 can obtain the importance score for each frame by interpolating the importance score for each segment.

[0134] In embodiments of this disclosure, the highlight keyword determination module 840 can determine highlight keywords based on importance distribution information. The highlight keyword determination module 840 can determine highlight keywords based on importance distribution information using the highlight keyword determination method described in this disclosure.

[0135] In embodiments of this disclosure, the highlight keyword determination module 840 may determine highlight keywords based on the highlight keyword priority database 860. In embodiments of this disclosure, the highlight keyword priority database 860 may store a weight for each keyword. In embodiments of this disclosure, the highlight keyword priority database 860 may be stored in the memory of the electronic device 100 or may be obtained from an external electronic device. In embodiments of this disclosure, the highlight keyword determination module 840 may determine only the keywords included in the highlight keyword priority database 860 as highlight keywords.

[0136] The summary keyword determination module 840 can determine summary keywords by applying a weight to each keyword to the importance distribution information of each keyword. According to embodiments of this disclosure, the summary keyword determination module 840 can determine summary keywords based on the weighted importance distribution information using the summary keyword determination method described in this disclosure. For example, even when a keyword with a low weight has a high importance score in its distribution information, that keyword may not be selected as a summary keyword.

[0137] In embodiments of this disclosure, the highlight video generation module 850 can generate highlight videos corresponding to highlight keywords. In embodiments of this disclosure, the highlight video generation module 850 can generate highlight videos that include frames of video corresponding to highlight segments.

[0138] In embodiments of this disclosure, highlight segments may be determined by at least one of an importance distribution determination module 830, a highlight keyword determination module 840, or a highlight video generation module 850. Highlight segments may be determined based on importance scores. For example, highlight segments may include segments with high importance scores.

[0139] In embodiments of this disclosure, the highlight video generation module 850 can receive selection input for highlight keywords. The highlight keywords can be displayed via a display of the electronic device 100. The highlight keyword determination module 840 can receive input for selecting at least some of the keywords to be displayed. For example, the highlight keyword determination module 840 can receive selection input via an input interface. The highlight video generation module 850 can generate a highlight video corresponding to the input highlight keywords. The electronic device 100 can provide highlight videos tailored to the user's intent by generating highlight videos based on the input for selecting highlight keywords, and can provide the user with the ability to modify the highlight videos according to the user's intent.

[0140] Figure 8b This is a diagram illustrating a module included in an electronic device 100 according to an embodiment of the present disclosure.

[0141] In embodiments of this disclosure, the electronic device 100 may include a keyword extraction module 810, a segment sampling module 820, an importance distribution determination module 830, a generalization distribution determination module 835, a highlight keyword determination module 840, a general keyword determination module 845, and a highlight video generation module 850. References to the keyword extraction module 810, segment sampling module 820, importance distribution determination module 830, highlight keyword determination module 840, and highlight video generation module 850 will be omitted. Figure 8a The description is as follows.

[0142] In embodiments of this disclosure, the generalization distribution determination module 835 can obtain generalization distribution information corresponding to keywords. In embodiments of this disclosure, the generalization distribution determination module 835 can obtain generalization distribution information including generalization scores based on time or frames. In embodiments of this disclosure, the generalization distribution determination module 835 can determine the generalization score based on the frequency of keywords included in multiple groups within a video.

[0143] In embodiments of this disclosure, the generalization distribution determination module 835 can identify the number of frames including the first keyword among multiple frames included in a first group of multiple groups. The generalization distribution determination module 835 can also identify the number of groups including the first keyword among multiple groups. The generalization distribution determination module 835 can obtain generalization distribution information corresponding to the first keyword based on the number of frames including the first keyword and the number of groups including the first keyword. For example, the generalization distribution determination module 835 can determine a higher generalization score as the number of frames including the first keyword increases, and can also determine a higher generalization score as the number of groups including the first keyword increases. In embodiments of this disclosure, the generalization score can be determined based on Equation 4.

[0144] [Equation 4]

[0145] Here, k represents the keyword index, d represents the group index, and TF... k,d This represents the number of keywords included in the group, N represents the total number of groups in the video, and df k This indicates the number of groups that include the keyword.

[0146] In embodiments of this disclosure, the generalization distribution determination module 835 may obtain generalization distribution information based on the number of groups including the first keyword. For example, the generalization distribution determination module 835 may determine a higher generalization score as the number of groups including the first keyword increases, and a lower generalization score as the number of groups including the first keyword decreases.

[0147] In embodiments of this disclosure, the generalization distribution determination module 835 can obtain generalization distribution information based on the ratio between the number of frames in a first group including the first keyword and the total number of keywords included in the first group. The generalization distribution determination module 835 can determine the ratio between the number of frames in a first group including the first keyword and the total number of keywords included in the first group. For example, the generalization distribution determination module 835 can determine the TF in Equation 1. k,d The ratio between the number of keywords included in the first group d and the number of keywords in the first group d, and generalization distribution information can be obtained based on the determined ratio.

[0148] In embodiments of this disclosure, the generalization score may be determined based on Equation 5.

[0149] [Equation 5]

[0150] Here, c k,d This represents the confidence score for the keyword, and the other variables may have the same meaning as those in Equation 4.

[0151] In embodiments of this disclosure, the generalization distribution determination module 835 is not limited to Equations 4 to 5, and can determine a generalization score such that the generalization score increases as the number of keywords included in a group increases, and the generalization score increases as the number of groups including keywords increases.

[0152] In embodiments of this disclosure, the generalization distribution determination module 835 can obtain a confidence score indicating whether each frame included in the first group contains a first keyword. For example, the generalization distribution determination module 835 can obtain c from Equation 5. k,d In embodiments of this disclosure, reference can be made to... Figure 8aThe described method determines the confidence score. According to embodiments of this disclosure, the generalization distribution determination module 835 can obtain generalization distribution information corresponding to the first keyword based on multiple confidence scores, the number of frames including the first keyword, and the number of groups including the first keyword.

[0153] In embodiments of this disclosure, the generalization distribution determination module 835 can obtain generalization distribution information based on the generalization score for each segment. The generalization distribution determination module 835 can obtain the generalization score for each frame by interpolating the generalization score for each segment.

[0154] In embodiments of this disclosure, the general keyword determination module 845 can determine general keywords based on generalization distribution information. In embodiments of this disclosure, "general keywords" can include keywords identified in many segments of the video. When the generalization score meets a defined condition, the general keyword determination module 845 can determine the keyword as a general keyword. For example, when the maximum or average value of the generalization score is greater than a threshold, the general keyword determination module 845 can determine the keyword as a general keyword.

[0155] In embodiments of this disclosure, the importance distribution determination module 830 and the generalization distribution determination module 835 can be implemented as a single module. For example, a module including the importance distribution determination module 830 and the generalization distribution determination module 835 can determine importance distribution information and generalization distribution information based on the frequency of keywords. In embodiments of this disclosure, the summary keyword determination module 840 and the general keyword determination module 845 can be implemented as a single module. For example, a module including the importance distribution determination module 830 and the generalization distribution determination module 835 can determine summary keywords and general keywords based on importance distribution information and generalization distribution information.

[0156] Figure 9 This is a diagram illustrating multiple groups obtained by dividing a video according to an embodiment of the present disclosure.

[0157] In embodiments of this disclosure, electronic device 100 can divide a video into multiple groups. Electronic device 100 can divide the video into multiple groups based on a division interval. For example, electronic device 100 can divide the video into a first group 912 corresponding to segment 1, a second group 914 corresponding to segment 2, and a third group 916 corresponding to segment 3. The first group 912, the second group 914, and the third group 916 may include the same number of frames. Electronic device 100 can obtain importance distribution information and / or generalized distribution information based on the number of keywords included in the first group 912, the second group 914, and the third group 916.

[0158] In embodiments of this disclosure, the electronic device 100 can divide video into multiple groups using multiple partitioning intervals. For example, see reference... Figure 9 The electronic device 100 can divide the video into a first group 912, a second group 914, and a third group 916 at a first dividing interval. The electronic device 100 can divide the video into multiple groups, including a fourth group 922 and a fifth group 924, at a second dividing interval. The electronic device 100 can divide the video into multiple groups, including a sixth group 932 and a seventh group 934, at a third dividing interval.

[0159] In embodiments of this disclosure, the second partitioning interval may be half of the first partitioning interval, and the third partitioning interval may be half of the second partitioning interval. However, this disclosure is not limited thereto, and multiple partitioning intervals may be determined independently.

[0160] In embodiments of this disclosure, the electronic device 100 may obtain importance distribution information and / or generalized distribution information based on the number of keywords included in multiple groups divided by multiple partitioning intervals. The electronic device 100 may obtain importance distribution information and / or generalized distribution information for each partitioning interval.

[0161] In embodiments of this disclosure, electronic device 100 may determine summary keywords for each segmentation interval based on importance distribution information for each segmentation interval. Electronic device 100 may determine the final summary keywords based on the summary keywords for each segmentation interval.

[0162] In embodiments of this disclosure, electronic device 100 may determine final importance distribution information and / or final generalization distribution information based on importance distribution information and / or generalization distribution information for each partition interval. For example, electronic device 100 may determine final importance distribution information and / or final generalization distribution information based on the average or sum of the importance scores of the importance distribution information and / or the generalization scores of the generalization distribution information for each partition interval.

[0163] In embodiments of this disclosure, electronic device 100 can determine summary keywords by using the final importance distribution information. Electronic device 100 can also improve the reliability of summary keyword determination by using multiple partitioning intervals.

[0164] Figure 10 This is a diagram illustrating the process of generating a highlight video according to embodiments of the present disclosure.

[0165] In embodiments of this disclosure, electronic device 100 can generate highlight videos according to defined rules. For example, instead of Figure 2 In operations S240 to S260, the electronic device 100 can determine the highlight segment based on the highlight keywords and generate a highlight video based on the highlight segment.

[0166] In embodiments of this disclosure, the importance distribution determination module 830 can determine importance distribution information 1010 for each keyword. The importance distribution determination module 830 can determine the importance distribution information 1010 corresponding to each keyword in the video. For example, the importance distribution determination module 830 can obtain importance distribution information 1010 related to each of the following: kissing, love, cute, daily life, hugging, smiling, and walking.

[0167] In embodiments of this disclosure, the highlight keyword determination module 840 can determine highlight keywords. The highlight keyword determination module 840 can select some from a plurality of video keywords as highlight keywords. For example, the highlight keyword determination module 840 can determine "kissing," "love," and "daily life" as highlight keywords.

[0168] In embodiments of this disclosure, the highlight video generation module 850 can generate a highlight video based on highlight keywords. The highlight video generation module 850 can also generate a highlight video based on importance distribution information 1020 corresponding to the selected highlight keywords. For example, the highlight video generation module 850 can select all of the highlight keywords "kiss," "love," and "daily life," and can generate a highlight video based on the importance distribution information 1020 of "kiss," "love," and "daily life."

[0169] In embodiments of this disclosure, the highlight video generation module 850 can generate highlight videos according to defined criteria. The highlight video generation module 850 can select at least some of the highlighted keywords according to the defined criteria. The highlight video generation module 850 can generate highlight videos by using the selected highlighted keywords.

[0170] In embodiments of this disclosure, the highlight video generation module 850 may select at least some of the highlight keywords based on user preference information. User preference information may include keywords that the user was previously interested in and / or keywords used in previously generated highlight videos.

[0171] In embodiments of this disclosure, the highlight video generation module 850 may select at least some of the highlight keywords based on priority information. The priority information may include predefined keyword priorities. The highlight video generation module 850 may select some of the highlight keywords according to the priorities based on the priority information.

[0172] In embodiments of this disclosure, the highlight video generation module 850 can determine highlight segments corresponding to highlight keywords. These highlight segments can be determined based on importance scores from importance distribution information 1020. The highlight video generation module 850 can determine the final highlight segment 1030 by combining highlight segments. For example, the highlight video generation module 850 can determine the final highlight segment 1030 by combining highlight segments of selected highlight keywords to generate a highlight video.

[0173] The highlight video generation module 850 can generate a highlight video based on the final highlight segment 1030. The highlight video generation module 850 can generate a highlight video that includes frames from the final highlight segment.

[0174] Figure 11 This is a diagram illustrating the process of determining collection keywords and general keywords according to embodiments of the present disclosure.

[0175] In embodiments of this disclosure, the keyword extraction module 810 can obtain (e.g., extract or determine) keywords from the video. See also... Figure 11 For example, a video could include frames of a woman walking on a beach with her pet. Keywords could include "woman," "pet," and "beach."

[0176] The highlight keyword determination module 840 can determine highlight keywords from the keywords extracted by the keyword extraction module 810. For example, highlight keywords may include kissing, hugging, smiling, and love. Highlight keywords can be keywords that appear multiple times in certain segments of the video. For example, highlight keywords can be keywords that are only included in frames of certain segments of the video. Highlight keywords can be used to determine highlight segments or generate highlight videos.

[0177] The general keyword determination module 845 can determine general keywords from the keywords extracted by the keyword extraction module 810. For example, general keywords may include beach, pet, and woman. General keywords can be keywords that appear multiple times throughout a frame of a video. For example, a video may include a scene of a woman and a pet on a beach. General keywords can be used to search for videos. For example, because a general keyword appears throughout a segment of a video, it may be suitable for searching the video.

[0178] Figure 12 This is a diagram illustrating an electronic device 100 displaying a UI for highlighting keywords according to an embodiment of the present disclosure.

[0179] In embodiments of this disclosure, the electronic device 100 may display indicators for video. For example, the electronic device 100 may display a compilation video with indicators representing video.

[0180] In embodiments of this disclosure, electronic device 100 may display stored video. Electronic device 100 may display a UI 1210 indicating the stored video. See also... Figure 12 UI 1210 may include a region 1215 for displaying one or more stored videos. For example, electronic device 100 may display UI 1210, which includes a region for displaying an image representing a video or a portion of a video (e.g., a highlight video). UI 1210 may include a region for displaying the playback time of each video.

[0181] In embodiments of this disclosure, electronic device 100 may receive input for selecting one of the videos. Electronic device 100 may display the selected video. In embodiments of this disclosure, electronic device 100 may display an image representing a video or highlight video. For example, electronic device 100 may display a UI 1220 indicating the video selected in response to the input. UI 1220 may include a region 1225 for displaying an image of the selected video or highlight video, and a region 1230 for displaying keywords corresponding to the displayed image or highlight video. For example, electronic device 100 may display a highlight video and may display highlight keywords corresponding to highlight segments or frames of the displayed highlight video.

[0182] In embodiments of this disclosure, electronic device 100 can send highlight videos to another electronic device. UI 1220 may include an indicator that directs the highlight videos to another electronic device. For example, UI 1220 may include a share icon 1235. A user can select highlight videos displayed on electronic device 100 and share the highlight videos with another electronic device.

[0183] Figure 13 This is a diagram illustrating a UI for editing video using an electronic device 100 according to an embodiment of the present disclosure.

[0184] In embodiments of this disclosure, electronic device 100 can send a highlight video to another electronic device based on keywords. Electronic device 100 can display a UI 1300 for editing the video. Electronic device 100 can display keywords for the video. For example, refer to... Figure 13 UI 1300 may include a region 1310 for displaying keywords. Electronic device 100 may display video corresponding to the keywords. For example, UI 1300 may include a region 1320 for displaying video corresponding to the keywords. Electronic device 100 may display segments of video corresponding to the keywords that are included in highlight videos. For example, UI 1300 may include a region 1330 that displays segments of video corresponding to the keywords that are included in highlight videos.

[0185] Electronic device 100 can generate highlight videos based on inputs corresponding to keywords. For example, electronic device 100 can generate a highlight video that includes a segment corresponding to "congratulations" based on input for a keyword corresponding to "congratulations". Electronic device 100 can also generate highlight videos based on the order of inputs. For example, based on input for "food" following input for "congratulations", electronic device 100 can generate a highlight video that includes a segment for "food" following the segment for "congratulations". In embodiments of this disclosure, electronic device 100 can send highlight videos generated using one or more keywords to another electronic device.

[0186] Figure 14 This is an illustration of a UI showing search results including images and / or videos according to embodiments of this disclosure.

[0187] In embodiments of this disclosure, electronic device 100 may indicate search results for images and / or videos based on keywords. Electronic device 100 may display a UI 1400 indicating search results for images and / or videos. For example, refer to... Figure 14 UI1400 may include a region 1410 for displaying found keywords and a region 1420 for displaying videos corresponding to the keywords.

[0188] In embodiments of this disclosure, electronic device 100 can search for images and / or videos based on compilation keywords and / or general keywords. Electronic device 100 can display the images and / or videos found based on compilation keywords and / or general keywords. For example, electronic device 100 can display only images and / or videos with keywords found as compilation keywords, only images and / or videos with keywords found as general keywords, or images and / or videos with keywords found as either compilation keywords or general keywords.

[0189] In embodiments of this disclosure, the electronic device 100 may display the type of keywords corresponding to the displayed images and / or videos. For example, the electronic device 100 may display an indicator indicating that the images and / or videos have found keywords as highlight keywords. For example, the electronic device 100 may display an indicator indicating that the images and / or videos have found keywords as general keywords. Here, the indicator may be displayed by icons, text, colors, or display area classification.

[0190] Figure 15 This is a diagram illustrating a UI for editing multiple videos by an electronic device 100 according to an embodiment of the present disclosure.

[0191] In embodiments of this disclosure, electronic device 100 can generate a highlight video based on multiple videos. In embodiments of this disclosure, electronic device 100 can display stored videos. Electronic device 100 can display a UI 1510 indicating the stored videos. See also... Figure 15 UI 1510 may include a region 1512 for displaying multiple stored videos. In embodiments of this disclosure, UI 1510 may correspond to Figure 12 UI 1210.

[0192] In embodiments of this disclosure, electronic device 100 may receive input for selecting multiple videos. Electronic device 100 may display indicators indicating the selected videos. For example, electronic device 100 may display checkmark icons on the selected videos. Electronic device 100 may generate a highlight video based on the selected videos.

[0193] Electronic device 100 can display a first video from the selected videos. For example, electronic device 100 can display a UI 1520 for editing the first video. Electronic device 100 can display keywords of the first video. For example, refer to... Figure 15 UI 1520 may include a region 1522 for displaying keywords. Electronic device 100 may display a first video corresponding to the keywords. For example, UI 1520 may include a region 1524 for displaying the first video corresponding to the keywords. Electronic device 100 may display a segment including the first video corresponding to the keywords. For example, UI 1520 may include a region 1526 for displaying a segment including the first video corresponding to the keywords.

[0194] Electronic device 100 can generate highlight videos based on input corresponding to keywords. For example, electronic device 100 can generate a highlight video that includes highlight segments corresponding to "congratulations" based on input for a keyword corresponding to "congratulations". Electronic device 100 can generate highlight videos based on one or more keywords of a first video. Electronic device 100 can display another video. For example, UI 1520 may include a region 1528 for navigating to an editing UI for other videos.

[0195] Electronic device 100 may display a second video from the selected videos. For example, electronic device 100 may display a UI 1530 for editing the second video. Electronic device 100 may display keywords for the second video. For example, UI 1530 may include an area 1532 for displaying keywords. Electronic device 100 may display the second video corresponding to the keywords. For example, UI 1530 may include an area 1534 for displaying the second video corresponding to the keywords. Electronic device 100 may display a segment including the second video corresponding to the keywords. For example, UI 1530 may include an area 1536 for displaying the segment including the second video corresponding to the keywords.

[0196] Electronic device 100 can generate highlight videos based on input corresponding to keywords. For example, electronic device 100 can generate a highlight video that includes a highlight segment corresponding to "high five" based on input for a keyword corresponding to "high five". Electronic device 100 can also generate highlight videos based on one or more keywords from a second video.

[0197] Figure 16 This is an illustration of a UI for sharing highlight videos by an electronic device 100 based on keywords, according to an embodiment of the present disclosure.

[0198] In embodiments of this disclosure, electronic device 100 can send a highlight video to another electronic device based on keywords. Electronic device 100 can display a UI 1600 for editing the video. Electronic device 100 can display keywords from the video. For example, refer to... Figure 16 UI 1600 may include a region 1610 for displaying keywords. Electronic device 100 may display video corresponding to the keywords. Electronic device 100 may display segments of video corresponding to the keywords that are included in highlight videos.

[0199] Electronic device 100 can receive input corresponding to multiple keywords. For example, electronic device 100 can receive input corresponding to "congratulations" and "food". Electronic device 100 can send a highlight video generated based on the multiple keywords corresponding to the input to another electronic device. For example, UI 1600 may include, for example, a share icon 1620. Users can select the highlight videos displayed on electronic device 100 and share the highlight videos with another electronic device.

[0200] Figure 17 This is an illustration of a UI showing search results including images and / or videos according to embodiments of this disclosure.

[0201] In embodiments of this disclosure, electronic device 100 may indicate search results for images and / or videos based on keywords. Electronic device 100 may display a UI 1700 indicating search results for images and / or videos. For example, refer to... Figure 17 UI1700 may include a region 1710 for displaying found keywords and a region 1720 for displaying videos corresponding to the keywords.

[0202] In embodiments of this disclosure, electronic device 100 may search for images and / or videos based on summary keywords and / or general keywords. Electronic device 100 may display images and / or videos found based on summary keywords and / or general keywords. Electronic device 100 may display videos containing keywords found as summary keywords and / or general keywords, or videos containing keywords similar to the found keywords as summary keywords and / or general keywords.

[0203] In embodiments of this disclosure, the electronic device 100 may display the type of keywords corresponding to the displayed images and / or videos. For example, the electronic device 100 may display an indicator indicating that the images and / or videos have found keywords as highlight keywords. For example, the electronic device 100 may display an indicator indicating that the images and / or videos have found keywords as general keywords. For example, the electronic device 100 may display an indicator indicating that the images and / or videos have keywords similar to the found keywords as general keywords. For example, the electronic device 100 may display a video with the keyword "love," which is similar to the found keyword "heart," as a highlight keyword.

[0204] Figure 18 This is a block diagram illustrating the configuration of an electronic device according to an embodiment of the present disclosure.

[0205] In embodiments of this disclosure, electronic device 100 may include processor 1810 and memory 1820.

[0206] Processor 1810 can control all operations of electronic device 100. For example, processor 1810 may be configured to execute one or more instructions of a program stored in memory 1820 to control all operations for electronic device 100 to generate highlight video. In embodiments of this disclosure, processor 1810 may include multiple processors. For example, multiple processors may execute one or more instructions of a program stored in memory 1820 individually or jointly to control all operations for electronic device 100 to generate highlight video.

[0207] In embodiments of this disclosure, processor 1810 may include configurations for controlling a series of processes, causing electronic device 100 to operate according to embodiments described in this disclosure. Processor 1810 may include one or more processors. The one or more processors included in processor 1810 may include circuitry such as a system-on-a-chip (SoC), integrated circuit (IC), etc. Processor 1810 may include one or more processors, each processor including a central processing unit (CPU), microprocessor unit (MPU), application processor (AP), digital signal processor (DSP), graphics processing unit (GPU), vision processing unit (VPU), application-specific integrated circuit (ASIC), programmable logic device (PLD), field-programmable gate array (FPGA), neural processing unit (NPU), communication processor, and / or an AI-specific processor designed with a hardware architecture for processing AI models, but not limited thereto.

[0208] Although not in Figure 18 As shown in the figure, the electronic device 100 may also include additional configurations for performing the operations described in the above embodiments. For example, the electronic device 100 may also include a display, camera, microphone, speaker, input / output interface, etc.

[0209] When the method according to embodiments of this disclosure includes multiple operations, the multiple operations may be executed by a single processor or by multiple processors. For example, when the first, second, and third operations are performed by the method according to embodiments of this disclosure, all of the first, second, and third operations may be executed by a first processor, or alternatively, the first and second operations may be executed by a first processor (e.g., a general-purpose processor), and the third operation may be executed by a second processor (e.g., an AI-specific processor). Here, an AI-specific processor, as an example of a second processor, may perform operations for training / inference of an AI model. However, embodiments of this disclosure are not limited thereto.

[0210] One or more processors 1810 according to this disclosure may be implemented as single-core processors or multi-core processors.

[0211] When a method according to an embodiment of the present disclosure includes multiple operations, the multiple operations may be performed by a single core, or may be performed by multiple cores included in one or more processors.

[0212] At least one processor 1810 according to embodiments of the present disclosure may include various types of processing circuitry and / or multiple processors. For example, the term "processor" as used herein (including in the claims) may include various types of processing circuitry comprising at least one processor, wherein one or more of the at least one processor are configured to perform the various functions described herein individually and / or collectively in a distributed manner. As used herein, when "processor," "at least one processor," and "one or more processors" are described as being configured to perform various functions, such terms may include, for example, but not limited to, cases where a single processor performs some of the described functions and other processors perform other functions of the described functions, and there are cases where a single processor can perform all of the described functions. Furthermore, at least one processor may include a combination of processors performing various listed / disclosed functions in a distributed manner. At least one processor may execute program instructions to implement or perform various functions.

[0213] The memory 1820 can store instructions, data structures, and program code that can be read by the processor 1810. The operations performed by the processor 1810 can be achieved by executing the instructions or code of the program stored in the memory 1820.

[0214] The memory 1820 may include at least one of volatile memory and non-volatile memory. For example, the memory 1820 may include at least one of flash memory, hard disk memory, multimedia card micro-memory, card-type memory (e.g., SD or XD memory), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, random access memory (RAM), or static random access memory (SRAM).

[0215] Memory 1820 may store one or more instructions and / or programs that cause electronic device 100 to operate to generate highlight videos. For example, memory 1820 may store instructions and / or programs for implementing operations for generating highlight videos. Processor 1810 may write data to memory 1820 or may read data stored in memory 1820. Processor 1810 may execute programs or at least one instruction stored in memory 1820 to process data according to predefined operating rules or artificial intelligence models. Processor 1810 may perform the operations described in embodiments of this disclosure.

[0216] The memory 1820 may also store instructions and / or programs for implementing at least one of the following functions: keyword extraction module, segment sampling module, importance distribution information generation module, generalized distribution information generation module, highlight keyword determination module, general keyword determination module, or highlight video generation module.

[0217] In embodiments of this disclosure, at least one instruction stored in memory 1820, when executed individually or jointly by at least one processor 1810, enables an electronic device to obtain one or more keywords included in multiple frames of a video. At least one instruction, when executed individually or jointly by at least one processor 1810, enables the electronic device to obtain importance distribution information corresponding to each of the one or more keywords. At least one instruction, when executed individually or jointly by at least one processor 1810, enables the electronic device to determine at least one highlight keyword among the one or more keywords based on the importance distribution information. At least one instruction, when executed individually or jointly by at least one processor 1810, enables the electronic device to display at least one highlight keyword. At least one instruction, when executed individually or jointly by at least one processor 1810, enables the electronic device to obtain input corresponding to at least a portion of the displayed at least one highlight keyword. At least one instruction, when executed individually or jointly by at least one processor 1810, enables the electronic device to generate a highlight video based on the highlight keyword corresponding to the input.

[0218] In embodiments of this disclosure, in addition to the processor 1810 and memory 1820, the electronic device 100 may also include other configurations. In embodiments of this disclosure, reference will now be made to… Figure 19 The configuration that the electronic device 100 may include is described in detail.

[0219] Figure 19 This is a block diagram illustrating the configuration of an electronic device according to an embodiment of the present disclosure.

[0220] like Figure 19 As shown, in addition to the processor 1810 and memory 1820, the electronic device 100 according to embodiments of the present disclosure may also include a camera 1910, a sensor unit 1920, a communication interface 1930 and / or a user interface 1940.

[0221] Processor 1810 controls the operation of electronic device 100. Processor 1810 can execute programs stored in memory 1820 to control camera 1910, sensor unit 1920, communication interface 1930, user interface 1940 and memory 1820.

[0222] The memory 1820 may store programs for processing and control by the processor 1810, and may store input / output data (e.g., video keywords, highlight keywords, or general keywords). The memory 1820 may also store artificial intelligence models. For example, the memory 1820 may store an artificial intelligence model for obtaining keywords.

[0223] Camera 1910 may refer to a device that acquires at least one frame. Here, at least one frame may be represented as an image (still image or video) or a picture.

[0224] Camera 1910 may include a wide-angle and / or telephoto camera capable of capturing images of the front or rear of electronic device 100. Camera 1910 may include a miniature camera or a pinhole camera. In embodiments of this disclosure, electronic device 100 may include multiple cameras 1910.

[0225] Camera 1910 may include an image sensor 1911 and / or an image signal processor (ISP) 1912. Image sensor 1911 may include means for identifying light transmitted through a lens of camera 1910. Image signal processor 1912 may refer to a processor that processes the signal identified by image sensor 1911. In embodiments of this disclosure, image signal processor 1912 may perform primary correction for distortion caused by camera 1910.

[0226] Sensor unit 1920 may include, but is not limited to, a depth sensor 1921 and / or an infrared sensor 1922. Depth sensor 1921 may include a sensor that measures depth information of an object. Infrared sensor 1922 may include a sensor that measures values ​​using infrared light. Infrared sensor 1922 can measure depth information of an object using infrared light.

[0227] Sensor unit 1920 can acquire depth information by using an ultra-wideband (UWB) sensor. The UWB sensor measures the round-trip time of the UWB signal. Sensor unit 1920 can determine depth information based on the round-trip time of the UWB signal.

[0228] The communication interface 1930 may include one or more components for performing communication between the electronic device 100 and the server device or between the electronic device 100 and the mobile terminal. For example, the communication interface 1930 may include a short-range communication unit 1931, a long-range communication unit 1932, etc.

[0229] The short-range communication unit 1931 may include a Bluetooth communication unit, a Bluetooth Low Energy (BLE) communication unit, a Near Field Communication (NFC) unit, a Wi-Fi (WLAN) communication unit, a ZigBee communication unit, an Infrared Data Association (IrDA) communication unit, a Wi-Fi Direct (WFD) communication unit, an Ultra Wideband (UWB) communication unit, or an Ant+ communication unit, but this disclosure is not limited thereto.

[0230] Long-range communication unit 1932 may include the Internet, computer networks (e.g., LAN or WAN), and mobile communication units. The mobile communication unit transmits and receives wireless signals from at least one of a base station, external terminal, or server in the mobile communication network. For example, the wireless signals may include various types of data based on the transmission and reception of voice call signals, video call signals, or text / multimedia messages. The mobile communication unit may include third-generation (3G) modules, fourth-generation (4G) modules, fifth-generation (5G) modules, Long Term Evolution (LTE) modules, Narrowband Internet of Things (NB-IoT) modules, Long Term Evolution for Machines (LTE-M) modules, etc., but this disclosure is not limited thereto.

[0231] User interface 1940 may include output interface 1941 and input interface 1942. Output interface 1941 may output audio or video signals and may include, for example, a display and / or a sound output unit.

[0232] In embodiments of this disclosure, the display can be configured as a touchscreen by forming a layer structure with a touchpad. When the display and touchpad form a layer structure and are configured as a touchscreen, the display can function as both an output interface 1941 and an input interface 1942. The display may include at least one of a liquid crystal display, a thin-film transistor liquid crystal display, a light-emitting diode (LED), an organic light-emitting diode, a flexible display, a 3D display, or an electrophoretic display. Furthermore, depending on the implementation of electronic device 100, electronic device 100 may include two or more displays. For example, electronic device 100 may include a forward-facing display and a rearward-facing display opposite to the forward orientation.

[0233] According to embodiments of this disclosure, the display can display and output information processed by the electronic device 100. For example, the display can display images captured in real time by the camera 1910 of the electronic device 100, or it can display images, videos, and / or highlight videos stored in the memory 1820. For example, the display can display references Figures 12 to 17 The UI is described. The display can output interfaces for controlling the electronic device 100, interfaces for displaying the status of the electronic device 100, etc.

[0234] The sound output unit can output audio data received from the communication interface 1930 or stored in the memory 1820. Furthermore, the sound output unit can output sound signals related to the functions performed by the electronic device 100. For example, the sound output unit may include a speaker or a buzzer. For example, the speaker or buzzer can output signals related to the functions performed by the electronic device 1000 as sound (e.g., call signal reception sound, message reception sound, or notification sound).

[0235] Input interface 1942 can receive input from a user. Input interface 1942 may include at least one of a keyboard, a dome switch, a touchpad (a capacitive touchpad, a pressure-resistive touchpad, an infrared beam sensing touchpad, a surface acoustic wave touchpad, an integrated strain gauge touchpad, a piezoelectric touchpad, etc.), a scroll wheel, a scroll wheel switch, and a microphone, but this disclosure is not limited thereto.

[0236] In embodiments of this disclosure, a method for generating highlight videos based on keywords, executed by an electronic device, is provided. The method may include: obtaining one or more keywords included in multiple frames of a video. The method may include: obtaining importance distribution information corresponding to each of the one or more keywords. The method may include: determining at least one highlight keyword among the one or more keywords based on the importance distribution information. The method may include: displaying at least one highlight keyword. The method may include: obtaining input corresponding to at least a portion of the displayed at least one highlight keyword. The method may include: generating a highlight video based on the highlight keyword corresponding to the input.

[0237] In embodiments of this disclosure, obtaining one or more keywords included in a video may include: obtaining a first frame from a plurality of frames in the video. Obtaining one or more keywords included in a video may include: obtaining feature information included in the first frame. The feature information may include information indicating at least one of an emotion, object, action, behavior, situation, location, or event included in the first frame. Obtaining one or more keywords included in a video may include: obtaining one or more keywords including keywords corresponding to the feature information.

[0238] In embodiments of this disclosure, one or more keywords may include a first keyword. The operation of obtaining importance distribution information corresponding to each of the one or more keywords may include: dividing the video into multiple groups based on a partitioning interval. The operation of obtaining importance distribution information corresponding to each of the one or more keywords may include: identifying the number of frames including the first keyword among multiple frames included in a first group of the multiple groups. The operation of obtaining importance distribution information corresponding to each of the one or more keywords may include: identifying the number of groups including the first keyword among the multiple groups. The operation of obtaining importance distribution information corresponding to each of the one or more keywords may include: obtaining importance distribution information corresponding to the first keyword based on the number of frames including the first keyword and the number of groups including the first keyword.

[0239] In embodiments of this disclosure, obtaining importance distribution information corresponding to the first keyword may include: determining the total number of keywords included in a first group of the video. Obtaining importance distribution information corresponding to the first keyword may include: determining a first ratio between the number of frames in the first group containing the first keyword and the total number of keywords included in the first group. Obtaining importance distribution information corresponding to the first keyword may include: determining a second ratio between the number of groups containing the first keyword and the total number of all groups included in the video. Obtaining importance distribution information corresponding to the first keyword may include: obtaining importance distribution information corresponding to the first keyword based on the first ratio and the second ratio.

[0240] In embodiments of this disclosure, the operation of obtaining importance distribution information corresponding to the first keyword may include: obtaining multiple confidence scores indicating whether each frame in a first group includes the first keyword. The operation of obtaining importance distribution information corresponding to the first keyword may include: obtaining importance distribution information corresponding to the first keyword based on multiple confidence scores, the number of frames, and the number of groups.

[0241] In embodiments of this disclosure, one or more keywords may include second keywords. The importance distribution information corresponding to the second keywords may include importance scores for the second keywords based on video frames or time. Determining at least one highlight keyword may include: determining that at least one highlight keyword includes the second keyword based on the second keyword's importance score satisfying a defined condition. Determining at least one highlight keyword may include: determining that at least one highlight keyword does not include the second keyword based on the second keyword's importance score not satisfying a defined condition.

[0242] In embodiments of this disclosure, the operation of generating a highlight video may include: determining highlight segments based on highlight keywords corresponding to the input. The operation of generating a highlight video may also include: generating a highlight video comprising multiple frames of the video corresponding to the highlight segments.

[0243] In embodiments of this disclosure, the input may include a first input corresponding to a first highlight keyword and a second input corresponding to a second highlight keyword. The operation of determining a highlight segment may include: determining a first highlight segment based on first importance distribution information corresponding to the first highlight keyword. The operation of determining a highlight segment may include: determining a second highlight segment based on second importance distribution information corresponding to the second highlight keyword. The operation of determining a highlight segment may include: determining a highlight segment that includes both the first and second highlight segments.

[0244] In embodiments of this disclosure, the first importance distribution information may include importance scores for first highlight keywords based on video frames or time. The first highlight segment may include frames with the highest importance score for the first highlight keywords and frames with importance scores for the first highlight keywords greater than a threshold.

[0245] In embodiments of this disclosure, the method may include: displaying a highlight video. The method may also include: displaying highlight keywords corresponding to highlight segments of the displayed highlight video.

[0246] In embodiments of this disclosure, an electronic device for generating highlight videos based on keywords is provided. The electronic device may include at least one processor comprising processing circuitry and a memory storing at least one instruction. When executed individually or jointly by the at least one processor, the at least one instruction causes the electronic device to: obtain one or more keywords included in a plurality of frames of a video. When executed individually or jointly by the at least one processor, the at least one instruction causes the electronic device to: obtain importance distribution information corresponding to each of the one or more keywords. When executed individually or jointly by the at least one processor, the at least one instruction causes the electronic device to: determine at least one highlight keyword among the one or more keywords based on the importance distribution information. When executed individually or jointly by the at least one processor, the at least one instruction causes the electronic device to: display at least one highlight keyword. When executed individually or jointly by the at least one processor, the at least one instruction causes the electronic device to: obtain input corresponding to at least a portion of the displayed at least one highlight keyword. When executed individually or jointly by the at least one processor, the at least one instruction causes the electronic device to: generate a highlight video based on the highlight keyword corresponding to the input.

[0247] In embodiments of this disclosure, at least one instruction, when executed individually or jointly by at least one processor, causes an electronic device to perform the following operations: acquire a first frame of a plurality of frames of video; and acquire feature information included in the first frame. The feature information may include information indicating at least one of emotion, object, action, behavior, situation, location, or event included in the first frame. At least one instruction, when executed individually or jointly by at least one processor, causes the electronic device to perform the following operations: acquire one or more keywords including keywords corresponding to the feature information.

[0248] In embodiments of this disclosure, one or more keywords may include a first keyword. At least one instruction, when executed individually or jointly by at least one processor, causes the electronic device to: divide a video into multiple groups based on a partitioning interval. At least one instruction, when executed individually or jointly by at least one processor, causes the electronic device to: identify the number of frames including the first keyword among multiple frames included in a first group of the multiple groups. At least one instruction, when executed individually or jointly by at least one processor, causes the electronic device to: identify the number of groups including the first keyword among the multiple groups. At least one instruction, when executed individually or jointly by at least one processor, causes the electronic device to: obtain importance distribution information corresponding to the first keyword based on the number of frames including the first keyword and the number of groups including the first keyword.

[0249] In embodiments of this disclosure, at least one instruction, when executed individually or jointly by at least one processor, causes the electronic device to perform the following operations: determine the total number of keywords included in a first group of videos. At least one instruction, when executed individually or jointly by at least one processor, causes the electronic device to perform the following operations: determine a first ratio between the number of frames in the first group including the first keyword and the total number of keywords included in the first group. At least one instruction, when executed individually or jointly by at least one processor, causes the electronic device to perform the following operations: determine a second ratio between the number of groups including the first keyword and the total number of groups included in the video. At least one instruction, when executed individually or jointly by at least one processor, causes the electronic device to perform the following operations: obtain importance distribution information corresponding to the first keyword based on the first and second ratios.

[0250] In embodiments of this disclosure, at least one instruction, when executed individually or jointly by at least one processor, enables the electronic device to perform the following operations: obtain multiple confidence scores indicating whether each frame in a plurality of frames of a first group includes a first keyword. At least one instruction, when executed individually or jointly by at least one processor, enables the electronic device to perform the following operations: obtain importance distribution information corresponding to the first keyword based on the multiple confidence scores, the number of frames, and the number of groups.

[0251] In embodiments of this disclosure, one or more keywords may include a second keyword. Importance distribution information corresponding to the second keyword may include an importance score for the second keyword based on video frames or time. At least one instruction, when executed individually or jointly by at least one processor, may cause the electronic device to: determine that at least one highlight keyword includes the second keyword based on the second keyword's importance score satisfying a defined condition. At least one instruction, when executed individually or jointly by at least one processor, may cause the electronic device to: determine that at least one highlight keyword does not include the second keyword based on the second keyword's importance score not satisfying a defined condition.

[0252] In embodiments of this disclosure, at least one instruction, when executed individually or jointly by at least one processor, enables the electronic device to perform the following operations: determine a highlight segment based on highlight keywords corresponding to the input. At least one instruction, when executed individually or jointly by at least one processor, enables the electronic device to perform the following operations: generate a highlight video comprising multiple frames of video corresponding to the highlight segment.

[0253] In embodiments of this disclosure, the input may include a first input corresponding to a first highlight keyword and a second input corresponding to a second highlight keyword. At least one instruction, when executed individually or jointly by at least one processor, may cause the electronic device to perform the following operations: determine a first highlight segment based on first importance distribution information corresponding to the first highlight keyword. At least one instruction, when executed individually or jointly by at least one processor, may cause the electronic device to perform the following operations: determine a second highlight segment based on second importance distribution information corresponding to the second highlight keyword. At least one instruction, when executed individually or jointly by at least one processor, may cause the electronic device to perform the following operations: determine a highlight segment including the first highlight segment and the second highlight segment.

[0254] In embodiments of this disclosure, the first importance distribution information may include importance scores for first highlight keywords based on video frames or time. The first highlight segment may include frames with the highest importance score for the first highlight keywords and frames with importance scores for the first highlight keywords greater than a threshold.

[0255] In embodiments of this disclosure, at least one instruction, when executed individually or jointly by at least one processor, causes the electronic device to perform the following operation: display a highlight video. At least one instruction, when executed individually or jointly by at least one processor, causes the electronic device to perform the following operation: display highlight keywords corresponding to highlight segments of the displayed highlight video.

[0256] In embodiments of this disclosure, a computer-readable recording medium may be provided having a program recorded thereon for performing any of the methods described above and below to operate an electronic device.

[0257] Embodiments of this disclosure can also be implemented as recording media (e.g., program modules including computer-readable instructions that will execute in a computer). Computer-readable recording media may include any available media accessible by a computer, volatile and non-volatile media, as well as removable and non-removable media. Computer-readable recording media may include computer storage media and communication media. Computer storage media includes all volatile and non-volatile media, as well as removable and non-removable media, and is technically implemented to store information including computer-readable instructions, data structures, program modules, or other data. Communication media includes computer-readable instructions, data structures, program modules, or other data (such as modulated data signals).

[0258] Computer-readable recording media according to embodiments of the present disclosure may be provided in the form of non-transitory recording media. Here, the term "non-transitory storage medium" may mean that the storage medium is a tangible device and does not include signals (e.g., electromagnetic waves), and may mean that data can be stored permanently or temporarily in the storage medium. For example, a "non-transitory storage medium" may include a buffer for temporarily storing data.

[0259] The methods according to embodiments of this disclosure may be included and set in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., an optical disc read-only memory (CD-ROM)), or may be distributed online through an app store (e.g., downloaded or uploaded) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable application) may be at least temporarily stored or temporarily generated in a machine-readable storage medium (such as the memory of a manufacturer's server, an app store's server, or a relay server).

[0260] Although this disclosure has been specifically shown and described with reference to the accompanying drawings, in which embodiments of the disclosure are illustrated, it will be apparent to those skilled in the art that the disclosure can be readily implemented in many different forms without altering the technical concept or essential characteristics of the disclosure. Therefore, it should be understood that the embodiments described herein should be considered descriptive only and not for limiting purposes. For example, singular configuration elements can be implemented in a distributed manner, and distributed configuration elements can also be combined and then implemented.

[0261] The scope of this disclosure is defined by the appended claims (rather than by the foregoing detailed descriptions), and all differences and modifications that may be derived from the meaning and scope of the claims and from other equivalent embodiments thereof shall be construed as being included in this disclosure.

Claims

1. A method for generating a highlight video based on keywords, executed by an electronic device, the method comprising: Obtain one or more keywords included in multiple frames of the video (S210). Obtain importance distribution information corresponding to each of the one or more keywords (S220). Based on the importance distribution information, at least one summary keyword is determined from the one or more keywords (S230). Display at least one of the highlights keywords (S240); Obtain input corresponding to at least a portion of the displayed at least one highlight keyword (S250). as well as Based on the highlight keywords corresponding to the input, a highlight video is generated (S260).

2. The method as described in claim 1, wherein, The operation of obtaining one or more keywords included in the video (S210) includes: Obtain the first frame of the plurality of frames in the video; Obtain feature information included in the first frame, wherein the feature information includes information indicating at least one of emotion, object, action, behavior, situation, location, or event included in the first frame; and Obtain the one or more keywords, wherein the one or more keywords include keywords corresponding to the feature information.

3. The method according to any one of claims 1 to 2, wherein, The one or more keywords include the first keyword, and The operation of obtaining importance distribution information corresponding to each of the one or more keywords (S220) includes: The video is divided into multiple groups based on the partitioning interval; Identify the number of frames containing the first keyword in the first group of the plurality of groups; Identify the number of groups that include the first keyword among the plurality of groups; and Based on the number of frames containing the first keyword and the number of groups containing the first keyword, importance distribution information corresponding to the first keyword is obtained.

4. The method of claim 3, wherein, The operations for obtaining importance distribution information corresponding to the first keyword include: Determine the total number of keywords included in the first group of the video; Determine a first ratio between the number of frames in the first group that include the first keyword and the total number of keywords included in the first group; Determine a second ratio between the number of groups including the first keyword and the total number of all groups included in the video; and Based on the first ratio and the second ratio, importance distribution information corresponding to the first keyword is obtained.

5. The method according to any one of claims 3 to 4, wherein, The operations for obtaining importance distribution information corresponding to the first keyword include: Obtain multiple confidence scores, wherein the multiple confidence scores indicate whether each frame in the first group of multiple frames includes the first keyword; and Based on the multiple confidence scores, the number of frames, and the number of groups, importance distribution information corresponding to the first keyword is obtained.

6. The method according to any one of claims 1 to 5, wherein, The one or more keywords include the second keyword. The importance distribution information corresponding to the second keyword includes the importance score of the second keyword based on the frame or time of the video, and The operation of determining the at least one highlight keyword (S230) includes: Based on the importance score of the second keyword satisfying the defined condition, it is determined that the at least one collection keyword includes the second keyword; and Based on the fact that the importance score of the second keyword does not meet the defined condition, it is determined that the at least one collection keyword does not include the second keyword.

7. The method according to any one of claims 1 to 6, wherein, The operation of generating the highlight video (S260) includes: Based on the keywords in the compilation corresponding to the input, the compilation segment is determined; and Generate the highlight video, wherein the highlight video includes multiple frames of the video corresponding to the highlight segments.

8. The method of claim 7, wherein, The input includes a first input corresponding to the keywords of the first compilation and a second input corresponding to the keywords of the second compilation, and The operation of determining the highlight segment includes: Based on the first importance distribution information corresponding to the first collection keywords, the first collection segment is determined; Based on the second importance distribution information corresponding to the keywords of the second collection, the second collection segment is determined; and The highlight segment is determined, wherein the highlight segment includes the first highlight segment and the second highlight segment.

9. The method of claim 8, wherein, The first importance distribution information includes importance scores for the first highlight keywords based on frames or time in the video, and The first compilation segment includes frames with the highest importance score of the first compilation keyword and frames with the importance score of the first compilation keyword greater than a threshold.

10. The method of any one of claims 7 to 9, further comprising: Display the aforementioned highlight video; as well as The highlights video displays the highlights section and the corresponding highlights keywords.

11. An electronic device (100) for generating highlight videos based on keywords, the electronic device (100) comprising: At least one processor (1810) includes processing circuitry; as well as The memory (1820) stores at least one instruction. When the at least one instruction is executed individually or jointly by the at least one processor (1810), it causes the electronic device (100) to perform the following operations: Obtain one or more keywords contained in multiple frames of a video; Obtain importance distribution information corresponding to each of the one or more keywords; Based on the importance distribution information, at least one summary keyword is determined from the one or more keywords; Display at least one of the featured keywords; Obtain input corresponding to at least a portion of the displayed at least one highlight keyword; and A highlight video is generated based on the highlight keywords corresponding to the input.

12. The electronic device (100) as claimed in claim 11, wherein, When the at least one instruction is executed individually or jointly by the at least one processor (1810), it causes the electronic device (100) to perform the following operations: Obtain the first frame of the plurality of frames in the video; Obtain feature information included in the first frame, wherein the feature information includes information indicating at least one of an object, action, behavior, situation, location, or event included in the first frame; and Obtain the one or more keywords, wherein the one or more keywords include keywords corresponding to the feature information.

13. The electronic device (100) according to any one of claims 11 to 12, wherein, The one or more keywords include the first keyword, and When the at least one instruction is executed individually or jointly by the at least one processor (1810), it causes the electronic device (100) to perform the following operations: The video is divided into multiple groups based on the partitioning interval; Identify the number of frames containing the first keyword in the first group of the plurality of groups; Identify the number of groups that include the first keyword among the plurality of groups; and Based on the number of frames containing the first keyword and the number of groups containing the first keyword, importance distribution information corresponding to the first keyword is obtained.

14. The electronic device (100) as claimed in claim 13, wherein, When the at least one instruction is executed individually or jointly by the at least one processor (1810), it causes the electronic device (100) to perform the following operations: Determine the total number of keywords included in the first group of the video; Determine a first ratio between the number of frames in the first group that include the first keyword and the total number of keywords included in the first group; Determine a second ratio between the number of groups including the first keyword and the total number of all groups included in the video; and Based on the first ratio and the second ratio, importance distribution information corresponding to the first keyword is obtained.

15. A computer-readable recording medium having a program recorded thereon for performing the method according to any one of claims 1 to 10 on a computer.

Images