Method, system, computer program (unsupervised video segment clustering and boundary positioning)

Abstract

To provide a method and apparatus for video processing.SOLUTION: A plurality of videos are accessed. An index is generated based on one or more key frames and one or more video features extracted from the plurality of video. A user input is processed to identify user intentions. The index is filtered to identify a plurality of candidate key frames based on the user intentions. A recommended video is generated based on the plurality of candidate key frames, further comprising selecting one or more additional frames based on processing the plurality of candidate key frames by using one or more clustering operations, and aggregating the plurality of candidate key frames and the one or more additional frames.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] FIELD OF THE DISCLOSURE This disclosure relates to video processing, and more particularly, to indexing, clustering, and positioning video frames based on user input. [Background technology]

[0002] Videos are becoming increasingly important to a wide range of people. Millions of videos are uploaded across various platforms every day, and these may include short user-generated clips posted on social media platforms, as well as professionally generated videos created for entertainment, educational, and / or business purposes (such as surgical videos for medical instruction, how-to videos for product demonstrations, or software tutorials).

[0003] With the considerable increase in the volume of available video content, the ability to accurately identify similar video content based on a user query has become increasingly important to both content creators and users. For example, a factory technician who relies on instructional videos that may contain significant background noise from machinery may search through a large number of videos to find guidance in a particular technical field. Similarly, a physician may search through a vast collection of surgical videos, typically created without audio, to find a particular procedure, such as a craniotomy. In the field of software development, many demo videos (e.g., usage guidance videos) are created to teach customers how to use various features within the software. However, during a software launch, a customer may only be interested in videos related to a particular feature. Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional methods that rely on basic keyword tagging or content analysis do not adequately meet the demand for effective and accurate video content retrieval based on user queries. [Means for solving the problem]

[0005] One embodiment presented in the present disclosure provides a method comprising: accessing a plurality of videos; generating an index for the plurality of videos based on one or more key frames and one or more video features extracted from each respective video of the plurality of videos; processing a user input to identify a user intent; filtering the index to identify a plurality of candidate key frames based on the user intent; and generating a recommended video based on the plurality of candidate key frames, wherein the generating the recommended video further comprises selecting one or more additional frames based on processing the plurality of candidate key frames using one or more clustering operations, and aggregating the plurality of candidate key frames and the one or more additional frames. One advantage provided by such an embodiment is clustering similar video frames guided by user input.

[0006] In another embodiment, one or more of the following features may be included: In one embodiment, processing the user input to identify user intent may include converting the user input into text data; and using natural language processing (NLP) techniques to process the text data and predict the user intent. Such an embodiment increases the system's ability to accurately identify user intent, resulting in more efficient video clustering and aggregation.

[0007] In one embodiment, the user input may include at least one of a text query, an audio file, an image, or a video file, and the user intent is presented as a row vector. Such an embodiment of representing user intent as a row vector streamlines integration with a clustering algorithm, thereby improving the efficiency of identifying candidate keyframes.

[0008] In one embodiment, selecting one or more additional frames based on processing the candidate keyframes using one or more clustering operations may include, for each candidate keyframe from a respective candidate video, identifying the one or more additional frames from the respective candidate video upon determining that a similarity between each of the one or more additional frames and the candidate keyframe exceeds a defined threshold, and clustering the candidate keyframe with the one or more additional frames from the respective candidate video to generate a video clip. Such an embodiment ensures that the system captures all frames that are relevant to a user's intent, thereby generating a cohesive and continuous video clip rather than isolated frames.

[0009] In one embodiment, the one or more key frames for each respective video of the plurality of videos may be determined by: dividing each respective video into a plurality of frames; extracting the one or more video features from each respective frame of the plurality of frames; clustering the plurality of frames based on a similarity between different frames of the plurality of frames, where the similarity is determined based on the one or more video features extracted from each respective frame; and identifying one or more cluster centers as the one or more key frames to present each respective video. Such an embodiment of pre-identifying key frames for each video enables faster and more accurate searches aligned with user intent.

[0010] In one embodiment, the one or more video features extracted from each respective video may include at least one of global features or local features. Such an embodiment of generating an index for a plurality of videos based on one or more global or local video features provides a quick reference to important features of a video in response to a user's input, thereby facilitating the search process.

[0011] In one embodiment, the method may further comprise, prior to generating the recommended video based on the plurality of candidate keyframes, recommending one or more key video features from one or more video features extracted from each respective video based at least in part on user intent. In some embodiments, the one or more additional frames may be selected based on the one or more recommended key video features. In some embodiments, the one or more recommended key video features may include at least one of: (i) color, (ii) luminance, (iii) texture, (iv) grayscale, (v) object tracing, (vi) scale-invariant feature transform (SIFT), or (vii) image semantics data. Such an embodiment provides increased versatility and visibility, enabling a user to make an informed decision regarding key video features consistent with their preferences and intent.

[0012] In one embodiment, the method may further include, prior to generating the recommended video based on the plurality of candidate keyframes, modifying the plurality of candidate keyframes, where modifying the plurality of candidate keyframes may further include removing a candidate keyframe from the plurality of candidate keyframes when determining that the candidate keyframe is an outlier among the plurality of candidate keyframes; and adding a candidate keyframe from the one or more keyframes in the index when determining that the candidate keyframe is visually similar to the candidate keyframe from the plurality of candidate keyframes. Such an embodiment improves accuracy of video recommendations by refining candidate keyframes.

[0013] Other embodiments of the present disclosure provide a non-transitory computer readable medium storing computer program code that, when executed by operation of one or more computer processors, performs operations in accordance with one or more of the above methods, as well as a system comprising one or more computer processors and one or more memories storing one or more programs that, when executed by the one or more computer processors, perform operations in accordance with one or more of the above methods. [Brief description of the drawings]

[0014] So that the above-described features of the present disclosure can be understood in more detail, a more particular description of the present disclosure briefly summarized above can be made with reference to embodiments, some of which are illustrated in the accompanying drawings. It should be noted, however, that the accompanying drawings illustrate typical embodiments and therefore should not be considered limiting; other equally effective embodiments are contemplated.

[0015] [Figure 1] FIG. 1 illustrates an exemplary computing environment for execution of at least a portion of the computer code involved in performing the methodology of the present invention.

[0016] [Diagram 2] FIG. 1 illustrates an example environment for video processing in accordance with some embodiments of the present disclosure.

[0017] [Diagram 3] FIG. 2 illustrates an example workflow for user input guided video segmentation, indexing and clustering according to some embodiments of the present disclosure.

[0018] [Figure 4] FIG. 2 illustrates an example method for video searching and clustering based on user input, according to some embodiments of the present disclosure.

[0019] [Diagram 5] 1 is a flow diagram illustrating an example method for user-driven video segment clustering and boundary location according to some embodiments of the present disclosure.

[0020] To facilitate understanding, wherever possible, the same reference numbers have been used to designate identical elements that are common to each of the figures. It is contemplated that elements disclosed in one embodiment may be beneficially used in other embodiments without specific recitation. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0021] The description of various embodiments of the present disclosure has been presented for illustrative purposes, but is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used in this specification has been selected to best explain the principles of the embodiments, practical applications, or technical improvements to the technology found in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

[0022] Embodiments herein describe a method or system for unsupervised user-driven video segment clustering and boundary positioning. As used herein, "user-driven" may refer to the ability of a method or system to adapt and / or respond to a user's specific input or requirements. As used herein, "user-driven" video segment clustering may refer to a process of grouping / clustering similar video segments (also referred to in some aspects as video frames) guided by user input. As used herein, "user-driven" video segment boundary positioning may refer to a process of determining boundaries within grouped video segments (also referred to in some aspects as video frames) as dictated by user input.

[0023] For example, in one embodiment, the user input may be provided in a variety of formats, such as a text query, an audio file, an image, or a video file. The user input may be converted into text data and processed using Natural Language Processing (NLP) techniques to identify the user's intent. Based on the identified user intent, the system may filter the index created for the keyframes to identify a set of candidate keyframes. In one embodiment, the index created for the keyframes may include all keyframes extracted from each video along with their respective video features (e.g., Local Binary Pattern (LBP) values, RGB vectors, HSV vectors, etc.). In some embodiments, the set of candidate keyframes may be selected based on the consistency between those features and the user's specified intent. Once the candidate keyframes are identified, the system may proceed to perform a clustering operation on other frames from each candidate video guided by the user's intent. For example, in one embodiment, the system may apply one or more clustering algorithms to other frames (e.g., frames not initially identified as candidate keyframes) from each candidate video, treating the candidate keyframes as center points for potential clusters. By comparing the similarity between each of the other frames and its respective candidate keyframe based on one or more video features (e.g., local binary pattern (LBP) values, RGB vectors, HSV vectors, etc.), the system may determine whether each of the other frames should be grouped with its respective candidate keyframe. In some embodiments, the clustering process may be repeated for all frames in each candidate video, generating multiple clusters. Following this, the system may then proceed to identify video segment boundaries for each cluster of frames.For example, in one embodiment, the system may generate consecutive clips by removing frames that are outside each cluster and / or do not exceed a predefined similarity threshold, reordering the remaining frames in each cluster (e.g., based on a defined criterion or according to their original order in the video), and concatenating these ordered frames. Finally, the system may generate a recommended video by aggregating consecutive clips from each cluster, where the content of the recommended video closely aligns with the user's intent.

[0024] FIG. 1 illustrates an exemplary computing environment for the execution of at least some of the computer code involved in performing the methodology of the present invention.

[0025] The computing environment 100 includes an example of an environment for the execution of at least a portion of computer code associated with performing the methodology of the present invention, such as video generation code 180. In addition to the video generation code 180, the computing environment 100 includes, for example, a computer 101, a wide area network (WAN) 102, an end user device (EUD) 103, a remote server 104, a public cloud 105, and a private cloud 106. In this embodiment, the computer 101 includes a processor set 110 (including processing circuitry 120 and cache 121), a communications fabric 111, a volatile memory 112, persistent storage 113 (including an operating system 122, and the video generation code 180 as identified above), a peripheral device set 114 (including a user interface (UI) device set 123, storage 124, and an Internet of Things (IoT) sensor set 125), and a network module 115. The remote server 104 includes a remote database 130. The public cloud 105 includes a gateway 140, a cloud orchestration module 141, a set of host physical machines 142, a set of virtual machines 143, and a set of containers 144.

[0026] Computer 101 may take the form of a desktop computer, a laptop computer, a tablet computer, a smartphone, a smartwatch or other wearable computer, a mainframe computer, a quantum computer, or any other form of computer or mobile device now known or later developed that is capable of executing programs, accessing a network, or querying a database, such as remote database 130. As is well understood in the art of computer technology, and depending on the technology, execution of a computer-implemented method may be distributed among multiple computers and / or among multiple locations. However, in this presentation of computing environment 100, in order to make the presentation as concise as possible, the detailed discussion focuses on a single computer, specifically computer 101. Although computer 101 is not shown in the cloud in FIG. 1, it may be located in the cloud. However, computer 101 is not required to be in the cloud except to any extent that may be categorically indicated.

[0027] Processor set 110 includes one or more computer processors of any type now known or developed in the future. Processing circuitry 120 may be distributed across multiple packages, e.g., multiple coordinated integrated circuit chips. Processing circuitry 120 may implement multiple processor threads and / or multiple processor cores. Cache 121 is memory located within the processor chip package and is typically used for data or code that should be available for fast access by threads or cores executing on processor set 110. Cache memory is typically organized into multiple levels depending on relative proximity to the processing circuitry. Alternatively, some or all of the cache for a processor set may be located "off-chip." In some computing environments, processor set 110 may be designed to operate with qubits and perform quantum computing.

[0028] Computer readable program instructions are typically loaded onto the computer 101 to cause a sequence of operational steps to be executed by the processor set 110 of the computer 101, thereby realizing a computer-implemented method, whereby the instructions so executed instantiate the method specified in the flowcharts and / or descriptions of the computer-implemented methods contained in this document (collectively, the "methods of the present invention"). These computer readable program instructions are stored in various types of computer readable storage media, such as cache 121 and other storage media discussed below. The program instructions and associated data are accessed by the processor set 110 to control and direct the execution of the methods of the present invention. In the computing environment 100, at least a portion of the instructions for executing the methods of the present invention may be stored in the video generation code 180 in persistent storage 113.

[0029] Communications fabric 111 is the signal-conducting paths that allow various components of computer 101 to communicate with one another. Typically, this fabric is made up of switches and conductive paths, such as buses, bridges, switches and conductive paths that make up physical input / output ports, and the like. Other types of signal communication paths, such as fiber optic and / or wireless communication paths, may be used.

[0030] Volatile memory 112 may be any type of volatile memory now known or later developed. Examples include dynamic random access memory (RAM) or static RAM. Typically, volatile memory 112 is characterized by random access, although this is not required unless expressly indicated. In computer 101, volatile memory 112 is located in a single package and is internal to computer 101, although alternatively or additionally, volatile memory may be distributed across multiple packages and / or located external to computer 101.

[0031] Persistent storage 113 is any form of non-volatile storage for a computer, now known or later developed. The non-volatility of this storage means that the stored data is maintained regardless of whether power is supplied to computer 101 and / or directly to persistent storage 113. Persistent storage 113 may be read-only memory (ROM), but typically at least a portion of persistent storage allows data to be written, data to be deleted, and data to be rewritten. Some well-known forms of persistent storage include magnetic disks and solid-state storage devices. Operating system 122 may take several forms, such as various known proprietary operating systems or open source Portable Operating System Interface type operating systems that employ a kernel. The code included in video generation code 180 typically includes at least a portion of the computer code associated with performing the methods of the present invention.

[0032] The peripheral device set 114 includes a set of peripheral devices of the computer 101. Data communication connections between the peripheral devices and other components of the computer 101 may be implemented in various ways, such as Bluetooth connections, Near-Field Communication (NFC) connections, connections made by cable (such as Universal Serial Bus (USB) type cables), insertion type connections (e.g., secure digital (SD) cards), connections made over local area communication networks, and even connections made over wide area networks such as the Internet. In various embodiments, the UI device set 123 may include components such as display screens, speakers, microphones, wearable devices (such as goggles and smart watches), keyboards, mice, printers, touch pads, game controllers, and haptic devices. The storage 124 is external storage, such as an external hard drive, or insertable storage, such as an SD card. The storage 124 may be persistent and / or volatile. In some embodiments, the storage 124 may take the form of a quantum computing storage device for storing data in the form of qubits. In embodiments requiring computer 101 to have a large amount of storage (e.g., where computer 101 stores and manages large databases locally), this storage may be provided by peripheral storage devices designed to store very large amounts of data, such as a storage area network (SAN) shared by multiple, geographically distributed computers. IoT sensor set 125 consists of sensors that can be used in Internet of Things applications. For example, one sensor may be a thermometer and another sensor may be a motion detector.

[0033] The network module 115 is a collection of computer software, hardware, and firmware that enables the computer 101 to communicate with other computers over the WAN 102. The network module 115 may include hardware such as a modem or Wi-Fi signal transceiver, software for packetizing and / or depacketizing data for communication network transmission, and / or web browser software for communicating data over the Internet. In some embodiments, the network control functions and network forwarding functions of the network module 115 are performed on the same physical hardware device. In other embodiments (e.g., embodiments utilizing software-defined networking (SDN)), the control functions and forwarding functions of the network module 115 are performed on physically separate devices such that the control functions manage several different network hardware devices. Computer-readable program instructions for carrying out the methods of the present invention can be downloaded to the computer 101 from an external computer or external storage device, typically via a network adapter card or network interface included in the network module 115.

[0034] WAN 102 is any wide area network (e.g., the Internet) capable of communicating computer data over non-local distances by any technology now known or later developed for communicating computer data. In some embodiments, WAN 102 may be replaced and / or supplemented by a local area network (LAN) designed to communicate data between devices located in a local area, such as a Wi-Fi® network. WANs and / or LANs typically include computer hardware such as copper transmission cables, optical transmission fiber, wireless transmission, routers, firewalls, switches, gateway computers, and edge servers.

[0035] End-user device (EUD) 103 is any computer system used and controlled by an end user (e.g., a customer of the enterprise that operates computer 101) and may take any of the forms discussed above in connection with computer 101. EUD 103 typically receives useful and useful data from the operation of computer 101. For example, in a hypothetical case in which computer 101 is designed to provide recommendations to the end user, the recommendations would typically be communicated from network module 115 of computer 101 over WAN 102 to EUD 103. In this manner, EUD 103 can display or otherwise present the recommendations to the end user. In some embodiments, EUD 103 may be a client device such as a thin client, a heavy client, a mainframe computer, a desktop computer, and the like.

[0036] Remote server 104 is any computer system that provides at least some data and / or functionality to computer 101. Remote server 104 may be controlled and used by the same entity that operates computer 101. Remote server 104 represents a machine that collects and stores useful and useful data for use by other computers, such as computer 101. For example, in the hypothetical case where computer 101 is designed and programmed to provide recommendations based on historical data, this historical data may be provided to computer 101 from remote database 130 of remote server 104.

[0037] A public cloud 105 is any computer system available for use by multiple entities that provides on-demand availability of computer system resources and / or other computer capabilities, particularly data storage (cloud storage) and computing power, without direct active management by users. Cloud computing typically leverages the sharing of resources to achieve coherence and economies of scale. Direct active management of the computing resources of the public cloud 105 is performed by computer hardware and / or software of a cloud orchestration module 141. The computing resources provided by the public cloud 105 are typically implemented by virtual computing environments running on various computers that make up the computers of the host physical machine set 142, which is the universe of physical computers within and / or available in the public cloud 105. A virtual computing environment (VCE) typically takes the form of virtual machines from the virtual machine set 143 and / or containers from the container set 144. It is understood that these VCEs may be stored as images and may be transferred among and between various hosts of physical machines either as images or after instantiation of the VCE. Cloud orchestration module 141 manages the transfer and storage of images, deploys new instantiations of VCEs, and manages active instantiations of VCE deployments. Gateway 140 is a collection of computer software, hardware, and firmware that enables public cloud 105 to communicate over WAN 102.

[0038] Some further description of a Virtualized Computing Environment (VCE) is now provided. A VCE can be stored as an "image." A new active instance of a VCE can be instantiated from an image. Two well-known types of VCE are virtual machines and containers. A container is a VCE that uses operating system level virtualization. This refers to a feature of an operating system where the kernel allows the existence of multiple isolated user space instances called containers. These isolated user space instances typically behave as real computers from the perspective of the programs running within them. A computer program running on a normal operating system can utilize all the resources of that computer, such as connected devices, files and folders, network shares, CPU power, and quantifiable hardware capabilities. However, a program running inside a container can only use the contents of the container and of the devices assigned to the container, a feature known as containerization.

[0039] A private cloud 106 is similar to a public cloud 105, except that the computing resources are only available for use by a single enterprise. Although the private cloud 106 is shown as being in communication with the WAN 102, in other embodiments, the private cloud may be completely disconnected from the Internet and only accessible via a local / private network. A hybrid cloud is a composite of multiple clouds of different types (e.g., private, community or public cloud types), often each implemented by a different vendor. Each of the multiple clouds remains a separate and distinct entity, but the larger hybrid cloud architecture is bound together by standardized or proprietary technologies that enable orchestration, management, and / or data / application portability between the constituent clouds. In this embodiment, both the public cloud 105 and the private cloud 106 are part of a larger hybrid cloud.

[0040] 2 illustrates an example environment 200 for video processing according to some embodiments of the present disclosure. In the illustrated example, the environment 200 includes one or more computing devices 201 for video processing, a database 240, and one or more user devices 245-1, 245-2. In some embodiments, one or more of the illustrated devices may be physical devices or systems. In other embodiments, one or more of the illustrated devices may be implemented using virtual devices and / or across multiple devices.

[0041] In the illustrated example, the computing device 201, the database 240, and the user devices 245-1, 245-2 are remote from each other and communicatively coupled to each other via the network 210. That is, the computing device 201, the database 240, and the user devices 245-1, 245-2 may each be implemented using separate hardware systems. The network 210 may include or correspond to a wide area network (WAN), a local area network (LAN), the Internet, an intranet, or any combination of suitable communication media that may be available, and may include wired, wireless, or a combination of wired and wireless links. In some embodiments, the computing device 201, the database 240, and the user devices 245-1, 245-2 are local to each other (e.g., within the same local network and / or the same hardware system) and may communicate with each other using any suitable local communication media, such as a local area network (LAN) (including a wireless local area network (WLAN)), hardwire, wireless links, an intranet, or the like.

[0042] In the illustrated example, database 240 includes a plurality of videos that may be in a variety of formats. For example, database 240 may store short videos (e.g., vlogs, commercial advertisements) and / or long videos (e.g., movies, documentaries, television programs). The content in database 240 may include user-generated videos (e.g., social media users) and / or videos produced by professional companies. Database 240 may also store videos created for a wide variety of purposes, such as education (e.g., surgical videos, university lectures, seminars, speeches, online courses, corporate training videos), entertainment (e.g., movies, television programs, documentaries, music videos, vlogs), and / or business promotion (e.g., commercial advertisements, product demonstration videos, usage guidance videos). The content of the stored videos may vary significantly, including, but not limited to, animation, live action, 3D videos, slow motion, time lapse, 360-degree immersive, and mixed media, among other formats.

[0043] In the illustrated example, the user devices 245-1, 245-2 act as an interface between the user and the computing device 201. The user devices 245-1, 245-2 may be used by the user to interact with the computing device 201, for example, by sending the user's input (which may include the user's preferences, requirements, search criteria, topics / concepts of interest, and the like) to the computing device 201, receiving output from the computing device 201, and / or displaying the output to the user. In one embodiment, the output of the computing device 201 may include a full video generated by aggregating and / or sorting similar video frames. The output video may present a continuous sequence of scenes that are consistent with the user's input. In some embodiments, the computing device 201 may return clusters of similar video frames or clips rather than a single continuous video, providing the user with the option to select content that matches their input criteria.

[0044] In some embodiments, the user devices 245-1, 245-2 may be responsible for prompting feedback from the users. For example, after reviewing the output from the computing device, depending on the type of output, the user may rate the output video, save related clips, and / or adjust the requirements or search criteria in the user input. The client devices may send feedback to the computing device 201 to further refine and personalize the video processing operations.

[0045] User input may be provided in a variety of formats. In one embodiment, a user may use a user device to enter specific search terms or select certain preferences and send a (e.g., text) query to computing device 201 for further processing. In some embodiments, a user may provide input by sending audio commands or uploading images, audio files, or video files to computing device 201.

[0046] In some embodiments, user devices 245-1, 245-2 may take a variety of forms, such as a desktop computer, a laptop computer, a tablet computer, a smartphone, a smartwatch, or other device that can be used to interface with computing device 201 over network 210.

[0047] In the illustrated example, the computing device 201 is capable of accessing, retrieving, and editing videos stored in the database 240. Although shown as a physical device, in embodiments, the computing device 201 may be implemented using virtual devices and / or across multiple devices (e.g., in a cloud environment). The computing device 201 may be connected to the user devices 245-1, 245-2 via a network 210, allowing for two-way communication. Through the network 210, the computing device 201 may receive user inputs from the user devices 245-1, 245-2. The computing device 201 may process these inputs to perform operations such as video segmentation, indexing, clustering, and / or boundary positioning, and send output back to the user devices for display.

[0048] In the illustrated example, the computing device 201 may store the generated results or analysis in the database 240 and / or the local storage 215. For example, in one embodiment, the computing device 201 may divide the video into individual frames and process each frame to extract one or more video features. The features may include characteristics of the video such as color, texture, grayscale, number of objects depicted, object descriptions, etc., among others. In some embodiments, the computing device 201 may generate metadata for each frame based on these video features. The metadata (also referred to as frame meta in some embodiments) (e.g., 315 in FIG. 3) may include one or more video features and their corresponding values ​​for each frame and serve as an overall reference for further comparison and analysis. The information in the metadata may include, but is not limited to, a video file identifier (e.g., file_id), a transformed frame identifier (e.g., frame_id), a texture pattern, a number of objects being rendered (e.g., num_obj), a description of the objects identified in the frame (also referred to in some embodiments as an image semantic description) (e.g., obj_des), a local binary pattern (LBP) value (e.g., LBP_val), a red, green, blue (RGB) vector or value (e.g., RGB_vec), a hue, saturation, value (HSV) vector or value (HSV_vec), and / or a hash value (e.g., hash_array) of the frame. The computing device 201 may store the individual frames along with their corresponding metadata in the database 240 and / or local storage 215. This operation may enable fast and efficient search and retrieval of both visual content (e.g., video frames) and analysis data (e.g., metadata) when new user input is received, without incurring substantial computational expense.

[0049] In some embodiments, the computing device 201 may extract keyframes based on similarity in metadata and index the extracted keyframes for further efficient searching and retrieval. The computing device 201 may store the extracted keyframes for each video along with the index in the database 240 and / or local storage 215 for further comparison and analysis when user input is received.

[0050] In some embodiments, the computing device 201 may process user input received from user devices (e.g., 245-1, 245-2) to identify user intent (which may include user preferences, requirements, search criteria, topics / concepts of interest, and the like). Based on the identified user intent, the computing device 201 may filter the index to identify a set of candidate keyframes based on corresponding metadata associated with each frame. The computing device 201 may use the candidate keyframes as center points to apply a clustering algorithm to other frames (e.g., frames not identified as candidate keyframes), which may include both keyframes and non-keyframes. The clustering algorithm may determine whether each of the other frames is sufficiently similar to a given candidate frame (e.g., center point) based on the metadata of each frame, where sufficiently similar frames may be grouped together. Using the clustering operation, the computing device may generate multiple clusters, where each cluster represents a group of video frames that are sufficiently similar to one of the candidate keyframes and likely also closely related to the user's specified intent. In some embodiments, the computing device 201 may store the generated clusters in the database 240 and / or local storage 215 for further processing.

[0051] In some embodiments, the computing device 201 may perform boundary positioning for each cluster of frames. For example, the computing device 201 may remove frames outside a defined threshold distance of each cluster center, reorder frames within a threshold distance of each cluster center, and / or generate a continuous clip for each individual cluster. In some embodiments, frames in a particular cluster (e.g., having a particular candidate keyframe as a center point) are from the same video. In some embodiments, the clustering operation may be performed at least in part on a video file identifier (e.g., file_id). That is, when assessing the similarity between different frames, the video file identifier may be considered to ensure that frames in the same cluster are from the same video as their respective center points (e.g., candidate keyframes). In some embodiments, after the clustering operation is completed, the identified frames in each cluster may be reordered according to their timestamps or chronological order in the original video. The sorted sequence of frames may then be used to generate a continuous video clip. The reordering process ensures that each of the clips maintains the original sequence of context from the source video.

[0052] In some embodiments, the computing device 201 combines these generated clips to create a full video and provides it as a response to the user input. In some embodiments, the full video may include multiple consecutive video clips, each generated from a respective cluster of frames. In some embodiments, these individual video clips may all be derived from the same video, and thus the full video may represent different segments of the original video. In some embodiments, these individual video clips may come from different videos, and the created full video may combine segments from various video sources. In some embodiments, the response may be sent back to the user device (e.g., 245-1, 245-2) over the network 210, allowing the user to view a generated video that accurately reflects their specified intent in their input. In some embodiments, in addition to being sent back to the user device for immediate viewing, the response may be stored in the database 240 and / or local storage 215 for future reference. The practice of storing responses may aid in further analysis and response generation (and substantially reduce computational costs) when similar user input is received, thus optimizing the system's operation for recurring user input. Additionally, in some embodiments, users may provide feedback about responses, such as rating the responses and / or rating the relevance of a video. The user's feedback, along with the stored responses, may then be used to further improve the accuracy and efficiency of the system.

[0053] In the illustrated example, computing device 201 includes a CPU 205, memory 295, storage 215, one or more network interfaces 225, and one or more I / O interfaces 220. In the illustrated embodiment, CPU 205 retrieves and executes programming instructions stored in memory 295, as well as stores and retrieves application data resident in storage 215. CPU 205 generally represents a single CPU and / or GPU, multiple CPUs and / or GPUs, a single CPU and / or GPU with multiple processing cores, and the like. Memory 295 is included to generally represent random access memory. Storage 215 may be any combination of disk drives, flash-based storage devices, and the like, and may include fixed and / or removable storage devices, such as fixed disk drives, removable memory cards, cache, optical storage, network attached storage (NAS), or storage area networks (SAN).

[0054] In some embodiments, I / O devices 235 (such as a keyboard, monitor, etc.) are connected via I / O interface 220. Additionally, via network interface 225, computing device 201 can be communicatively coupled to one or more other devices and components (e.g., database 240, user devices 245-1, 245-2) over network 210 (which may include the Internet, a local network, and the like). As shown, CPU 205, memory 295, storage 215, network interface 225, and I / O interface 220 are communicatively coupled by one or more interconnect buses 230.

[0055] In the illustrated embodiment, memory 295 includes frame generation module 250, feature extraction module 255, key frame identification module 260, indexing module 265, voice / text module 270, user intent analyzer 275, image semantics module 280, video clustering and normalization module 285, and video editing module 290. Though illustrated as separate components for conceptual clarity, in some embodiments the operations of the illustrated components (and other components not illustrated) may be combined or distributed across any number of components. Additionally, while illustrated as software resident within memory 295, in some embodiments the operations of the illustrated components (and other components not illustrated) may be implemented using hardware, software, or a combination of hardware and software.

[0056] In one embodiment, the frame generation module 250 may access the videos stored in the database 240 and split each video into individual frames (e.g., 310 in FIG. 3). In some embodiments, the frame generation module 250 may scan through each video file (e.g., 305 in FIG. 3) and decode each video file into a series of individual frames (e.g., 310 in FIG. 3). As used herein, an individual frame may refer to a still image that represents a particular instant in the video, and a series of individual frames may represent the progression of an image over the duration of the video. In some embodiments, each frame may be associated with a timestamp or timeline, which indicates a particular time or sequence in the video. In some embodiments, the timestamp or timeline is indicated in the frame identifier (e.g., frame_id) of each frame. In some embodiments, the frame generation module 250 may store the generated frames (e.g., 310 in FIG. 3) for each video in the database 240 or local storage 215 for further processing. In some embodiments, the frame generation module 250 may feed the generated frames (e.g., 310 in FIG. 3) directly to subsequent modules for feature extraction, keyframe identification, indexing, and / or clustering.

[0057] In one embodiment, the feature extraction module 255 may receive individual video frames (e.g., 310 in FIG. 3), for example from the frame generation module 250. The feature extraction module 255 may process each frame to identify and extract relevant video features using various image processing techniques. The relevant video features may include global and / or local features. In some embodiments, the global features may include color distribution (e.g., RBG, HSV), texture patterns (e.g., LBP), brightness and contrast, grayscale, and other visual elements that can be used to understand the overall properties of the frame. In some embodiments, the local features may include interest points (e.g., Scale Invariant Feature Transform (SIFT)), object shapes and boundaries, motion vectors, and other visual elements that can be used to characterize specific areas or objects within the frame.

[0058] In some embodiments, the feature extraction module 255 may generate metadata (also referred to in some embodiments as frame meta) for each frame (e.g., 315 in FIG. 3) based on the extracted video features. For example, the metadata may include a video file identifier (e.g., file_id), a transformed frame identifier (e.g., frame_id), a texture pattern, a number of objects (e.g., num_obj), a description of the objects identified in the frame (also referred to in some embodiments as image semantic description) (e.g., obj_des), an LBP value (e.g., LBP_val), an RGB vector (e.g., RGB_vec), an HSV vector (HSV_vec), and / or a hash value (e.g., hash_array). The metadata may include values ​​for each extracted feature, which serve as a quantitative representation of the visual content of the frames for further comparison and analysis. In some embodiments, the feature extraction module 255 may associate the generated metadata with its respective video frame and store it in the database 240 or local storage 215 for further processing. In some embodiments, the feature extraction module 255 may feed the generated metadata and / or its respective frames directly to subsequent modules for keyframe identification, indexing, and / or clustering.

[0059] In one embodiment, the key frame identification module 260 may use the metadata generated by the feature extraction module 255 to assess the similarity between different frames in the same video and identify key frames for the video. In some embodiments, the similarity may be determined based on one or more video features, including global and / or local features. One or more similarity measures may be used. For example, the similarity between different frames may be measured using cosine similarity, Euclidean distance, or a correlation measure where one or more video features are mapped to a shared vector space. Based on the measured similarity, the key frame identification module 260 may identify key frames (e.g., 320 in FIG. 3) for each video. In some embodiments, the key frame identification module 260 may identify key frames by applying a clustering algorithm to group similar frames together. For example, the key frame identification module 260 may apply a clustering algorithm to all frames in the same video to generate clusters of frames that share common visual characteristics. Once the clusters of frames are generated, the keyframe identification module 260 may select the most representative frame in each cluster (e.g., the center point of each cluster in feature space) as a keyframe for the respective video. In some embodiments, the keyframe identification module 260 may store the keyframes (e.g., 320 in FIG. 3) and their metadata for each video in the database 240 or local storage 215 for further processing. In some embodiments, the keyframe identification module 260 may feed the keyframes and their metadata directly to subsequent modules for indexing, clustering, and / or boundary location.

[0060] In one embodiment, the indexing module 265 may index the keyframes based on their metadata. The generated index (e.g., 325 in FIG. 3) may include the sequence or timestamp of each keyframe with respect to the video from which the keyframe was extracted (e.g., keyframe array) and / or the video feature values ​​contained in its respective metadata (e.g., video file identifier, frame identifier, texture pattern, number of objects, object description, LBP value, RGB vector, HSV vector, and / or hash value). In some embodiments, the indexing module 265 may store the index (e.g., 325 in FIG. 3) in the database 240 or local storage 215 for further processing. In some embodiments, the indexing module 265 may feed the index directly to a subsequent module for clustering and / or boundary location.

[0061] In one embodiment, the user intent analyzer 275 may process one or more user inputs received from a user. The user may provide the input using a user device (e.g., 245-1, 245-2). The user input may be provided in a variety of formats, including text, audio, image, or video. If the user input is not in text format (e.g., if the input includes an audio or video file), the user intent analyzer 275 may forward these inputs to the voice-to-text module 270, which transcribes the spoken language (e.g., audio commands, audio tracks in an audio or video clip) into written text. In some embodiments, a machine learning model may be used to perform the speech-to-text conversion. The machine learning model may be trained using a large amount of audio data as input and its corresponding text transcription as output. In some embodiments, the computing device 201 may further include a video / text module that detects and / or recognizes key objects within a user input (eg, a video clip, or an image) and represents them in textual form.

[0062] After all user inputs are converted into text format, in one embodiment, the user intent analyzer 275 may process the text data to infer the user intent. In the present case, the user intent may include various information such as the user's search criteria, preferences or requirements, the user's interest in a particular topic or concept, etc. Several techniques may be used to identify the user intent. For example, in some embodiments, the user intent analyzer 275 may utilize natural language processing (NLP) techniques to perform semantic analysis, which involves extracting keywords or phrases (e.g., representing key topics or concepts) in each input, understanding the relationship between the keywords or phrases to correctly interpret the meaning, and / or ranking the keywords or phrases based on their importance within each input. In some embodiments, the importance of the words or phrases may be measured by frequency of occurrence and / or position within each input. In some embodiments, the user intent analyzer 275 may represent the user intent in a machine-readable format, such as a row vector (e.g., 340 in FIG. 3 ) generated based on the ranked keywords or phrases.

[0063] In one embodiment, the identified user intents may then be provided to the index module 265. Using these intents, the index module 265 may filter through an index (e.g., 325 in FIG. 3) to identify a set of candidate keyframes (e.g., 345 in FIG. 3) that potentially match the user's specified intent.

[0064] In one embodiment, the image semantics module 280 may process the metadata and / or corresponding individual video frames to generate image semantics (e.g., 350 in FIG. 3) for each frame. The image semantics may represent a higher level description of the content in each video frame and may include one or more video features from the corresponding metadata in different combinations. For example, one set of image semantics may include text patterns, number of objects (e.g., num_obj), description of objects identified in the frame (also referred to in some embodiments as image semantic description) (e.g., obj_des), and LBP value (e.g., LBP_val), while another set of image semantics may include text patterns, number of objects (e.g., num_obj), LBP value (e.g., LBP_val), and HSV vector (e.g., HSV_vec). In some embodiments, the individual video features may serve as inputs in the process of performing video clustering. In other embodiments, the image semantics (e.g., a combination of two or more video features) may be used as a reference or input for a clustering operation. The clustering operation may involve applying a clustering algorithm to identify frames that are similar to candidate keyframes. By using the image semantics as a reference, the similarity between different frames may be measured based on one or more video features. In some embodiments, each video feature in the image semantics may be assigned a different weight based on its relative importance and relevance to the identified user intent.

[0065] In one embodiment, the feature clustering and normalization module 285 may receive the candidate keyframes and their corresponding metadata, as well as other frames and their corresponding metadata. The feature clustering and normalization module 285 first normalizes different video features to a common scale to produce optimal results. This normalization may be particularly useful when more than one video feature is used in comparing the similarity between different frames, because different video features (e.g., color, texture, or motion) may have very different scales and ranges. In some embodiments, the video features may be adjusted to a common scale between 0 and 1. The normalization may ensure that each video feature contributes equally to the model's judgment, thereby improving the efficiency and accuracy of the resulting clusters.

[0066] Once normalization is complete, in some embodiments, the feature clustering and normalization module 285 may apply a clustering algorithm to check whether other keyframes (e.g., not initially identified as relevant to the user's intent) should be grouped with the candidate keyframe. The feature clustering and normalization module 285 may use the candidate keyframe (identified by the index module 265) as the center point of the cluster, evaluate each of the other frames, and assign each of the other keyframes to the cluster (e.g., to the candidate keyframe) to which it is most similar according to the measured similarity. In some embodiments, the measured similarity used by the clustering algorithm may be based on a single video feature, such as color (e.g., RGB values), texture (e.g., LBP values), grayscale, object tracing (e.g., HSV values), scale (scale invariant feature transform (SIFT)), among others. In some embodiments, the measured similarity used by the clustering algorithm may be based on a combination of different video features, such as image semantics generated by the image semantics module 280. In some embodiments, the feature clustering and normalization module 285 may prioritize certain video features over others based on identified user intent and / or historical data and recommend the top features as key features when clustering other keyframes with candidate keyframes.

[0067] Based on the clustering results, in some embodiments, the feature clustering and normalization module 285 may generate a set of high-precision candidate keyframes (e.g., 355 in FIG. 2). This set may include both the candidate keyframes initially identified as relevant to the user's intent (e.g., 245 in FIG. 3) and additional keyframes determined to be relevant (e.g., within a defined threshold distance of the cluster center) based on the clustering algorithm. In some embodiments, the information about the high-precision candidate keyframes may then be updated to the index module 265. The index module 265 may integrate the optimized / adjusted information into the index, thereby enhancing its ability to more accurately identify candidate keyframes for future similar user intents. Once a high-precision candidate keyframe (e.g., 355 in FIG. 3) is identified, the feature clustering and normalization module 285 may run the clustering algorithm a second time (or run a new clustering algorithm) using the high-precision candidate keyframe as a new center point. An algorithm may then be used to evaluate all other non-keyframes to determine their similarity to these high-precision candidate keyframes (e.g., center points). In some embodiments, a predefined similarity threshold between the frames and their respective closest high-precision candidate keyframes (e.g., new center points) may be established to ensure that the selected non-keyframes are sufficiently relevant to the user's intent. Thus, non-keyframes that exhibit a high degree of visual similarity to the high-precision candidate keyframes and exceed the predefined similarity threshold are then grouped together. These selected non-keyframes, together with their corresponding high-precision candidate keyframes, may be referred to as candidate frames (e.g., 360 in FIG. 3), which are then processed to create a series of continuous video clips (e.g., 365 in FIG. 3).

[0068] In some embodiments, the feature clustering and normalization module 285 may identify a candidate keyframe as an outlier during the clustering process. For example, a candidate keyframe may be classified as an outlier if it is significantly different from the majority of other candidate keyframes and / or if other frames do not share significant similarity to this candidate keyframe (e.g., other frames should not be grouped with this candidate keyframe). Under these circumstances, the feature clustering and normalization module 285 may exclude this outlier keyframe from further calculations to ensure that the clustering results are not distorted and that the resulting video remains coherent and representative of the user's intent.

[0069] In some embodiments, the feature clustering and normalization module 285 may use the candidate keyframes (e.g., 345 in FIG. 3) as the center points of the clusters and apply a clustering algorithm to all other frames (including keyframes not initially identified as relevant to the user's intent, and non-keyframes) to check whether they are visually similar to the candidate keyframes (e.g., fall within their respective clusters) and sufficiently relevant to the user's intent (e.g., exceed a defined similarity threshold). That is, the system may perform a single clustering operation on the candidate keys to discover other associations, rather than the two-cluster operation discussed above.

[0070] In one embodiment, the video editing module 290 may process the candidate frames (e.g., 360 in FIG. 3) to generate a series of one or more consecutive video clips (e.g., 365 in FIG. 3). In some embodiments, the candidate frames may include the candidate key frames (e.g., 345 in FIG. 3) that were initially identified as relevant to the user's intent, additional key frames determined to be relevant based on the first clustering result, and additional non-key frames determined to be relevant based on the second clustering result. In some embodiments, the candidate frames may include the candidate key frames (e.g., 345 in FIG. 3) that were initially identified as relevant to the user's intent, and additional frames (including both key frames and non-key frames) determined to be relevant based on the single clustering result. In some embodiments, the video editing module 290 may stitch together the series of consecutive video clips to create a single full video and transmit it to the user device (e.g., 245-1, 245-2).

[0071] 3 illustrates an example workflow 300 for user input guided video segmentation, indexing, and clustering according to some embodiments of the present disclosure. In some embodiments, the workflow 300 may be performed by one or more computing devices, such as computer 101 as shown in FIG 1 and / or computing device 201 as shown in FIG 2. Although illustrated as separate components for conceptual clarity, in some embodiments, the operations of the illustrated components (and other components not shown) may be combined or distributed across any number of various components and may be implemented using hardware, software, or a combination of hardware and software.

[0072] In the illustrated example, one or more videos 305 stored in a database (e.g., 240 in FIG. 2) are provided to a frame generator module 250. Upon receiving the videos 305, the frame generator module 250 scans and divides each video into a series of individual frames 310. As discussed above, in some embodiments, an individual frame may refer to a still image that represents a particular moment in time within the video, and a series of individual frames may represent the progression of an image over the duration of the video. In some embodiments, each individual frame may be associated with a timestamp or time series, which indicates a particular time or sequence within the video.

[0073] In the illustrated example, the generated individual frames 310 are sent to the feature extraction module 255, which processes each frame to extract relevant video features. The relevant video features may include both global and / or local features. In some embodiments, the global features may include color distribution (e.g., RBG, HSV), texture patterns (e.g., LBP), brightness and contrast, grayscale, and other visual elements that can be used to understand the overall characteristics of the frame. In some embodiments, the local features may include interest points (e.g., Scale Invariant Feature Transform (SIFT)), object shapes and boundaries, motion vectors, and other visual elements that can be used to characterize specific areas or objects within the frame.

[0074] In the illustrated example, once the relevant video features for each frame have been extracted, the feature extraction module 255 then proceeds to generate metadata (also referred to as frame meta in some embodiments) (e.g., 315) for each frame. As discussed above, the metadata may include various information about the frame. For example, in some embodiments, the metadata may include a video file identifier (e.g., file_id), a transformed frame identifier (e.g., frame_id), a texture pattern, a number of objects (e.g., num_obj), a description of the objects identified in the frame (also referred to as image semantic description in some embodiments) (e.g., obj_des), an LBP value (e.g., LBP_val), an RGB vector (e.g., RGB_vec), an HSV vector (HSV_vec), and / or a hash value (e.g., hash_array). In the shown example, the generated metadata 315 (e.g., along with their respective frames 310) are provided to subsequent modules (e.g., keyframe identification module 260, image semantics module 280, or feature clustering and normalization module 285) for further processing, as discussed in more detail below.

[0075] In the illustrated example, the metadata 315 (e.g., along with their respective frames 310) are provided to a key frame identification module 260 to identify key frames 320 for each video. The key frame identification module 260 may evaluate similarities between different frames in the same video and identify key frames based on these similarities. In some embodiments, the similarities between different frames may be determined using one or more video features and their corresponding values ​​stored in the metadata of each frame.

[0076] In the illustrated example, the index module 265 receives the key frames 320 and generates an index 325 for all the key frames based on their metadata. The generated index 325 may include a sequence or timestamp of each key frame (e.g., a key frame array) and video feature values ​​stored in each key frame's respective metadata, including, but not limited to, a video file identifier (e.g., file_id), a transformed frame identifier (e.g., frame_id), a texture pattern, a number of objects (e.g., num_obj), a description of the objects identified in the frame (also referred to in some embodiments as an image semantic description) (e.g., obj_des), an LBP value (e.g., LBP_val), an RGB vector (e.g., RGB_vec), an HSV vector (HSV_vec), and / or a hash value (e.g., hash_array).

[0077] In the illustrated example, user input 330 is similarly received. If the received user input 330 is not in text format (e.g., the input includes an audio or video file), the user input may first be processed by a voice / text module 270, which converts audio data (e.g., audio commands, audio tracks within an audio or video clip) into written text (e.g., text input 335). In some embodiments, the voice / text module 270 may include one or more machine learning models that have been trained using large amounts of audio data and their corresponding text descriptions. In some embodiments, user input including a video clip or image may be provided to a video / text module, which detects and / or recognizes key objects within the video clip or image and represents them in text format.

[0078] In the illustrated example, the user intent analyzer 275 receives text input 335 and interprets the input to identify the user's intent. Several techniques may be used to identify the user's intent. As discussed above, in some embodiments, NLP techniques may be used by the user intent analyzer 275 to perform semantic analysis, including extracting keywords or phrases (e.g., representing key topics or concepts) in each input, identifying relationships between the keywords or phrases to interpret meaning, and ranking the keywords or phrases based on their importance within each input. As shown, the output of the user intent analyzer 275 is a row vector 340, which represents the user's intent based on the ranked keywords or phrases.

[0079] In the illustrated example, a vector 340 representing the user's intent is provided to an indexing module 265, which searches in an index 325 based on the vector 340 to identify a set of candidate keyframes 345. In some embodiments, the set of candidate keyframes may refer to keyframes that are deemed potentially related to or consistent with the user's intent. The identified candidate keyframes are then provided to a feature clustering and normalization module 285 for further processing.

[0080] In the illustrated example, the candidate keyframes 345 and their corresponding metadata, as well as the other frames 310 and their corresponding metadata, are provided by the feature clustering and normalization module 285. Upon receiving the frames and metadata, the feature clustering and normalization module 285 may first normalize the different video features in the metadata to a common scale. The normalization may ensure that each video feature contributes equally to the model's judgment, thereby improving the efficiency and accuracy of the resulting clusters. Once normalization is complete, the feature clustering and normalization module 285 may use the candidate keyframe as a center point and apply a clustering algorithm to all other keyframes in the database (e.g., those not initially identified as relevant to the user's intent) to check whether the other keyframes are visually similar to the candidate keyframe and therefore should be grouped together with the candidate keyframe.

[0081] Based on the clustering results, as shown, the feature clustering and normalization module 285 generates a set of high-precision candidate keyframes 355 as discussed above. In the example shown, the set of high-precision candidate keyframes 355 is updated to the indexing module 265 to improve its ability to more accurately identify candidate keyframes when similar user intents are received in the future.

[0082] In the illustrated example, the feature clustering and normalization module 285 runs the clustering algorithm a second time (or runs a new clustering algorithm) using the high-precision candidate keyframe 355 as the new center point. During the second run, the clustering algorithm may be applied to all other non-keyframes (e.g., frame 310) to determine their similarity to these high-precision candidate keyframes. Non-keyframes that show a high degree of visual similarity to the high-precision candidate keyframes and exceed a predefined similarity threshold (set to ensure consistency with user intent) are grouped together to create a set of candidate frames 360. In general, the clustering algorithm used to group the frames may include k-means clustering, mean-shift clustering, density-based spatial clustering of applications with noise (DBSCAN), hierarchical clustering, spectral clustering, or any other suitable clustering algorithm.

[0083] In the illustrated example, the set of candidate frames 360 are then sent to the video editing module 290, which creates a series of continuous video clips 365 for the candidate frames 360. The process of generating the continuous video clips 365 may include several stages. For example, in some embodiments, the video editing module 290 may first remove frames that are outside their respective clusters and / or are outliers whose similarity to their respective center points (e.g., high-precision candidate keyframes 355) does not exceed a predefined similarity threshold. This process ensures that only frames that are sufficiently relevant to the user's intent and visually close to the high-precision candidate keyframes 355 are included in the final video output. After the irrelevant frames are removed, the video editing module 290 may then proceed to reorder the remaining frames in each cluster (e.g., candidate frames 360) according to their timestamps or chronological order in the original video to maintain the continuity and narrative coherence of the original video. The video editing module 290 may then concatenate the reordered frames in each cluster to create a continuous video clip 365. Each video clip 365 may have a defined start and end point and represent a segment of the original video that matches the user's intent. In some embodiments, the process may be repeated for each cluster and each high accuracy candidate keyframe 355 to generate a set of continuous video clips 365 from different videos that are all consistent with the user's intent. In some embodiments, the continuous video clips may be provided to the user individually. In the example shown, the video editing module 290 may stitch these clips 365 together to create a full video and send it to the user device (e.g., 245-1, 245-2).

[0084] In some embodiments, visual similarity may be measured based on a single video feature, such as color (e.g., RGB values), texture (e.g., LBP values), grayscale, object tracing (e.g., HSV values), scale (Scale Invariant Feature Transform (SIFT)), among others. In some embodiments, the visual similarity used by the clustering algorithm may be based on a combination of different video features, such as the image semantics 350 generated by the image semantics module 280. As discussed above, the image semantics 350 may include one or more video features from the corresponding metadata in different combinations and may be used as a reference to compare the similarity between different frames. In some embodiments, the feature clustering and normalization module 285 may prioritize certain video features over others based on the identified user intent and / or historical data, and recommend the top features as key features when clustering other frames with candidate key frames. For example, if a user is looking for a guidance video for craniotomy surgery, the user input may include words or phrases such as "surgery," "craniotomy," "technique," and / or "procedure." By performing a relational semantic analysis on the user input, the user intent analyzer 275 may infer that the user is primarily interested in the detailed procedures and techniques used in the craniotomy. The user intent analyzer 275 may represent the intent in a vector 340 and send it to the feature clustering and normalization module 285. Based on the interpretation of the user input vector and / or the analysis of the historical data, the feature clustering and normalization module 285 may determine that the object tracing feature (which focuses on the detailed motion of the object in the frame) is more relevant and effective than other features to capture the precise details of the surgical process. Therefore, the feature clustering and normalization module 285 may recommend that the user select the object tracing feature as the main feature in the video clustering process.In some embodiments, the feature clustering and normalization module 285 may automatically apply the recommended key features in the video clustering process (rather than waiting for user approval or confirmation). In some embodiments, the historical data may include past user inputs, the specific video features used for each input, and the corresponding clustering outcomes that result from these features.

[0085] 4 illustrates an example method 400 for video searching and clustering based on user input according to some embodiments of the present disclosure. In some embodiments, the method 400 may be performed by one or more computing devices, such as the computer 101 as shown in FIG. 1 and / or the computing device 201 as shown in FIG. 2.

[0086] Method 400 begins at block 405, where a computing system accesses one or more videos. The videos may be stored on a remote data set (e.g., 240, FIG. 2) and may be accessed and retrieved by the computing system via a network connection (e.g., 210, FIG. 2), such as the Internet. Each of the videos may be divided into a series of individual frames (e.g., 310, FIG. 3) for further processing. A frame of a video may refer to a still image that forms part of a sequence of images that make up the video.

[0087] At block 410, the computing system analyzes each individual frame (e.g., 310 in FIG. 3) to identify and extract one or more relevant video features. The relevant video features may include global and / or local features. In some embodiments, the computing system may generate metadata (also referred to as frame meta in some embodiments) for each frame (e.g., 315 in FIG. 3) based on the extracted video features.

[0088] At block 415, the computing system calculates similarities between different frames in the same video and identifies key frames (e.g., 320 in FIG. 3) for each video. In some embodiments, key frames in a video may refer to frames that store important information or indicate critical moments in a video sequence and / or frames that represent substantial changes in content. Various methods may be used to identify key frames. For example, in some embodiments, the computing system may calculate similarities between different frames and apply a clustering algorithm to group similar frames together. Once groups / clusters of similar frames are generated, the system may select the most representative frame in each group / cluster (e.g., the center point of each cluster in the feature space) as a key frame for that video.

[0089] At block 420, the computing system indexes the keyframes (e.g., 320 in FIG. 3) for further filtering, clustering, and boundary positioning based on user input. The generated index (e.g., 325 in FIG. 3) may include the sequence or timestamp of each keyframe and / or the video feature values ​​contained within its respective metadata.

[0090] At block 425, the computing system receives user input (e.g., 330 in FIG. 3). The user input may be sent from a user device (e.g., 245-1, 245-2 in FIG. 2). The user input may be provided in a variety of formats. For example, the user input may include a text query, a natural language query, a keyword search, a Boolean search, among others. In some embodiments, the user input may include an audio file, such as a voice command, an audio recording, or a voice message provided by a user to interact with the device. In some embodiments, the user input may include an image. In some embodiments, the user input may include a video file, such as a meeting recording or a video message.

[0091] At block 430, the computing system determines whether the received user input (e.g., 330 of FIG. 3) is in text format. If the user input is in text format, the method proceeds to block 440. If the user input is not in text format, the method proceeds to block 435, where the computing system converts the user input to text format (e.g., 335 of FIG. 3). For example, in some embodiments, if the user input includes audio and / or video, the computing system may extract an audio track from the audio and / or video and transcribe the spoken language into written text. In some embodiments, if the user input includes video and / or images, the computing system may use object recognition techniques to identify key objects in the video and / or images and represent them in the written text.

[0092] In block 440, the computing system processes the text data format (e.g., 335 in FIG. 3) to infer the user's intent. In the present case, the user's intent may include a variety of information such as the user's search criteria, preferences or requirements, the user's interest in a particular topic or concept, etc. As discussed above, in some embodiments, NLP techniques may be used by the user intent analyzer 275 to perform semantic analysis, including extracting keywords or phrases (e.g., representing key topics or concepts) in each input, identifying relationships between the keywords or phrases to properly interpret the meaning, and ranking the keywords or phrases based on their importance in each input. In some embodiments, the importance of words or phrases may be measured by frequency of occurrence and / or position in each input. In some embodiments, the computing system may represent the user's intent in a machine-readable format, such as a row vector (e.g., 340 in FIG. 3) generated based on the ranked keywords or phrases. For example, a user may provide a text input of "I would like to know about asset criticality and health matrix capabilities for next year's matrices." The system may process the user input and identify the keywords as "asset," "criticality," "health," and "matrix." The system may then check existing dictionary words and place a "1" in the row vector if a word from the dictionary is present in the user's input. In the example given, the word "matrix" appears twice, while the words "asset," "criticality," and "health" each appear once. Thus, the row vector for the user input would be [1,...1,...1,...2,...].

[0093] In block 445, the computing system may filter the index based on the identified user intent. The computing system may identify a set of candidate keyframes (e.g., 345 in FIG. 3) that are associated with or consistent with the user intent.

[0094] At block 450, the computing system performs feature clustering. In some embodiments, the computing system may first use the candidate keyframe as a center point and apply a clustering algorithm to all keyframes. The first clustering process may check whether other keyframes (e.g., frames not initially identified as relevant to the user's intent) are visually similar to the candidate keyframe and therefore should be grouped together. If additional keyframes are identified as sufficiently similar to the candidate keyframe, the computing system may generate a set of high-precision candidate keyframes (e.g., 355 in FIG. 3). In some embodiments, the method 400 may return to block 420, where the computing system integrates optimized / adjusted information regarding the high-precision candidate keyframe into the index. After the high-precision candidate keyframe is identified, the computing system may proceed to run the clustering algorithm a second time (or run a new clustering algorithm) using the high-precision candidate keyframe as the new center point and apply the algorithm to all other non-keyframes. The computing system may assign each of the non-keyframes to the cluster (e.g., high-precision candidate keyframes) to which it is most similar based on the selected similarity measure. In some embodiments, a predefined similarity threshold between the frames and their respective center points (e.g., high-precision candidate keyframes) may be set up. The computing system selects non-keyframes as candidate frames only if they fall within the respective clusters and their similarity to the cluster center points exceeds the predefined similarity threshold.

[0095] At block 455, the computing system generates a video consistent with the user's intent based on the selected candidate frames (e.g., 360 in FIG. 3). In some embodiments, the candidate frames may include the candidate keyframes (e.g., 345 in FIG. 3) initially identified as relevant to the user's intent, additional keyframes determined to be relevant based on the first clustering results, and additional non-keyframes determined to be relevant based on the second clustering results.

[0096] FIG. 5 is a flow diagram illustrating an example method 500 for user-driven video segment clustering and boundary location according to some embodiments of the present disclosure.

[0097] The method 500 begins at block 505, where a system (eg, computer 101 of FIG. 1 or computing device 201 of FIG. 2) accesses a plurality of videos (eg, 305 of FIG. 3).

[0098] At block 510, the system generates an index (e.g., 325 in FIG. 3 ) for the multiple videos based on one or more keyframes (e.g., 320 in FIG. 3 ) and one or more video features extracted from each respective video of the multiple videos. In one embodiment, the one or more video features may include global features and / or local features. In some embodiments, the global features may include color distribution (e.g., RBG, HSV), texture patterns (e.g., LBP), brightness and contrast, grayscale, and other visual elements that can be used to understand the overall characteristics of the frame. In some embodiments, the local features may include interest points (e.g., scale-invariant feature transform (SIFT)), object shapes and boundaries, motion vectors, and other visual elements that can be used to characterize specific areas or objects within a frame.

[0099] At block 515, the system processes the user input (e.g., 330 in FIG. 3) to identify user intent. In one embodiment, the system may represent the user intent as a row vector (e.g., 340 in FIG. 3). In one embodiment, the system may first convert the user input into text data (e.g., 335 in FIG. 3) and use a trained machine learning model to process the text data and predict the user intent. In some embodiments, the user input may be provided in a variety of formats, such as a text query, an audio file, an image, or a video file.

[0100] At block 520, the system filters the index (e.g., 325 in FIG. 3) based on user intent to identify a number of candidate keyframes (e.g., 345 in FIG. 3). In one embodiment, the number of candidate keyframes (e.g., 345 in FIG. 3) correspond to one or more candidate videos in a number of videos (e.g., 305 in FIG. 3).

[0101] In block 525, the system generates a recommended video (e.g., 370 in FIG. 3) based on the plurality of candidate keyframes (e.g., 345 in FIG. 3). The process of generating the recommended video (e.g., 370 in FIG. 3) includes selecting one or more additional frames (e.g., 360 in FIG. 3) based on processing the plurality of candidate keyframes (e.g., 345 or 355 in FIG. 3) using one or more clustering operations, and aggregating the plurality of candidate keyframes and the one or more additional frames. In some embodiments, the process of selecting one or more additional frames based on processing the plurality of candidate keyframes using one or more clustering operations includes, for each candidate keyframe from each candidate video, identifying one or more additional frames from the respective candidate videos upon determining that a similarity between each of the one or more additional frames and the candidate keyframe exceeds a specified threshold, and clustering the candidate keyframe with the one or more additional frames from the respective candidate videos to generate a video clip (e.g., 365 in FIG. 3). In some embodiments, prior to generating the recommended video based on the plurality of candidate keyframes, the system may recommend one or more key video features from one or more video features extracted from each respective video based at least in part on user intent. The key video features may then be used to measure the similarity between each of the one or more additional frames and its respective candidate keyframe to determine whether they should be grouped together to generate a series of consecutive clips. In some embodiments, the key video features may include color, luminance, texture, grayscale, object tracing, scale invariant feature transform (SIFT), or image semantics data. In some embodiments, prior to generating the recommended video based on the plurality of candidate keyframes, the system may adjust the plurality of candidate keyframes to generate a set of high-precision candidate keyframes (e.g., 355 in FIG. 3). The system may remove a candidate keyframe if it is determined to be an outlier among other candidate keyframes.If the system determines that one or more keyframes are visually similar to a candidate keyframe, the system may add one or more keyframes (e.g., not initially identified as being associated with the user's intent). In some embodiments, the system may generate a continuous video clip (e.g., 365 in FIG. 3) by combining multiple candidate keyframes (e.g., 345 or 355 in FIG. 3) and one or more additional frames (e.g., 360 in FIG. 3). In some embodiments, the continuous video clip (e.g., 365 in FIG. 3) may have a defined start and end point and / or represent a segment of the original video that matches the user's intent.

[0102] In the foregoing, reference has been made to the embodiments presented in this disclosure. However, the scope of this disclosure is not limited to the specifically described embodiments. Instead, any combination of these features and elements, whether associated with different embodiments or not, is contemplated for implementing and practicing the contemplated embodiments. Furthermore, while the embodiments disclosed herein may achieve advantages over other possible solutions or over the prior art, whether or not a particular advantage is achieved by a given embodiment does not limit the scope of this disclosure. Thus, the aspects, features, embodiments, and advantages discussed herein are merely exemplary and should not be considered elements or limitations of the appended claims unless expressly recited in the claims. Similarly, references to "the present invention" should not be construed as a generalization of any of the inventive subject matter disclosed herein and should not be considered elements or limitations of the appended claims unless expressly recited in the claims.

[0103] Aspects of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware aspects, all of which may be generally referred to herein as a "circuit," "module," or "system."

[0104] Various aspects of the disclosure are described by way of text, flow charts, block diagrams of computer systems, and / or block diagrams of machine logic included in embodiments of computer program products (CPPs). For any flow chart, depending on the technology involved, operations may be performed in a different order than that shown in a given flow chart. For example, two operations shown in successive flow chart blocks may be performed in reverse order, as a single integrated step, simultaneously, or in an at least partially overlapping manner, also depending on the technology involved.

[0105] A computer program product embodiment ("CPP embodiment" or "CPP") is a term used in this disclosure to describe any set of one or more storage media (also referred to as "media") collectively contained in a set of one or more storage devices that collectively contain machine-readable code corresponding to instructions and / or data for performing computer operations specified in a given CPP claim. A "storage device" is any tangible device that can hold and store instructions for use by a computer processor. Without limitation, a computer-readable storage medium may be an electronic storage medium, a magnetic storage medium, an optical storage medium, an electromagnetic storage medium, a semiconductor storage medium, a mechanical storage medium, or any suitable combination of the foregoing. Some known types of storage devices that include these media include diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanically encoded devices (such as punch cards or pits / lands formed on the major surface of a disk), or any suitable combination of the foregoing. A computer-readable storage medium, as the term is used in this disclosure, is not to be construed as storage in the form of a transitory signal per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide, light pulses passing through fiber optic cables, electrical signals communicated through wires, and / or other transmission media. As will be appreciated by those skilled in the art, data typically moves at some indefinite time during normal operation of the storage device, such as during access, defragmentation, or garbage collection, but the above does not render the storage device transitory since the data is not transitory while it is stored.

[0106] While the forgoing is directed to embodiments of the present invention, other and further embodiments of the invention may be devised without departing from the basic scope thereof, the scope of the invention being determined by the claims which follow.

Claims

1. accessing a plurality of videos; generating an index for the plurality of videos based on one or more keyframes and one or more video features extracted from each respective video of the plurality of videos; processing the user input to identify user intent; filtering the index to identify a number of candidate keyframes based on the user intent; and generating a recommended video based on the plurality of candidate keyframes, selecting one or more additional frames based on processing the plurality of candidate keyframes using one or more clustering operations; and aggregating the plurality of candidate keyframes and the one or more additional frames. The stage A method comprising:

2. The step of processing the user input to identify user intent includes: converting the user input into text data; and using natural language processing (NLP) techniques to process the text data and predict the user intent.

2. The method of claim 1, comprising:

3. 10. The method of claim 1, wherein the user input comprises at least one of a text query, an audio file, an image, or a video file, and the user intent is represented as a row vector.

4. Selecting one or more additional frames based on processing the plurality of candidate key frames using one or more clustering operations includes: identifying, for each candidate keyframe from each candidate video, the one or more additional frames from the respective candidate video upon determining that a similarity between each of the one or more additional frames and the candidate keyframe exceeds a defined threshold; and clustering the candidate keyframes with the one or more additional frames from the respective candidate videos to generate a video clip. The method of claim 1 , comprising:

5. The one or more keyframes for each respective video of the plurality of videos include: Dividing each respective video into a number of frames; extracting the one or more video features from each respective frame of the plurality of frames; clustering the plurality of frames based on a similarity between different frames of the plurality of frames, where the similarity is determined based on the one or more video features extracted from each respective frame; and Identifying one or more cluster centers as the one or more keyframes to present each respective video. The method of claim 1 , wherein the

6. The method of claim 1 , wherein the one or more video features extracted from each respective video include at least one of global features or local features.

7. prior to generating the recommended videos based on the plurality of candidate keyframes, recommending one or more key video features from the one or more video features extracted from each respective video based at least in part on the user intent. The method of claim 1 further comprising:

8. The method of claim 7 , wherein the one or more additional frames are selected based on the one or more recommended key video features.

9. 8. The method of claim 7, wherein the one or more recommended key video features include at least one of: (i) color, (ii) luminance, (iii) texture, (iv) grayscale, (v) object tracing, (vi) scale-invariant feature transform (SIFT), or (vii) image semantics data.

10. Prior to generating the recommended video based on the plurality of candidate keyframes, modifying the plurality of candidate keyframes, removing a candidate keyframe from the plurality of candidate keyframes when determining that the candidate keyframe is an outlier among the plurality of candidate keyframes; and adding a keyframe of the one or more keyframes in the index if the keyframe is visually similar to a candidate keyframe of the plurality of candidate keyframes. The further step The method of claim 1 , further comprising:

11. one or more computer processors; and one or more memories that collectively store one or more programs that perform operations when executed by the one or more computer processors; A system comprising: The operation includes: Steps to access multiple videos: generating an index for the plurality of videos based on one or more keyframes and one or more video features extracted from each respective video of the plurality of videos; processing user input to identify user intent; filtering the index to identify a number of candidate keyframes based on the user intent; and A method for generating a recommended video based on the plurality of candidate keyframes, comprising: selecting one or more additional frames based on processing the plurality of candidate keyframes using one or more clustering operations; and aggregating the plurality of candidate keyframes and the one or more additional frames; A procedure having A system comprising:

12. The steps to process user input and identify user intent are: converting said user input into text data; and using natural language processing (NLP) techniques to process the text data and predict the user intent; The system of claim 11 , further comprising:

13. 12. The system of claim 11, wherein the user input comprises at least one of a text query, an audio file, an image, or a video file, and the one or more video features extracted from each respective video comprise at least one of global features or local features.

14. The step of selecting one or more additional frames based on processing the plurality of candidate key frames using one or more clustering operations comprises: for each candidate keyframe from each candidate video, identifying one or more additional frames from the respective candidate video upon determining that a similarity between each of the one or more additional frames and the candidate keyframe exceeds a defined threshold; and clustering the candidate keyframes with the one or more additional frames from the respective candidate videos to generate a video clip. The system of claim 11 , comprising:

15. The operation includes: and prior to generating the recommended videos based on the plurality of candidate keyframes, recommending one or more key video features from the one or more video features extracted from each respective video based at least in part on the user intent. The system of claim 11 further comprising:

16. The system of claim 15 , wherein the one or more additional frames are selected based on the one or more recommended key video features.

17. 16. The system of claim 15, wherein the one or more recommended key video features include at least one of: (i) color, (ii) luminance, (iii) texture, (iv) grayscale, (v) object tracing, (vi) scale-invariant feature transform (SIFT), or (vii) image semantics data.

18. The operation includes: Prior to the step of generating a recommended video based on the plurality of candidate keyframes, a step of modifying the plurality of candidate keyframes is provided, removing a candidate keyframe from the plurality of candidate keyframes when determining that the candidate keyframe is an outlier among the plurality of candidate keyframes; and adding a keyframe to the one or more keyframes in the index upon determining that the keyframe is visually similar to a candidate keyframe of the plurality of candidate keyframes; The method further comprises the steps of: The system of claim 11 further comprising:

19. 1. A computer program comprising computer readable program code that performs operations when executed by operation of one or more computer processors, The above operation is Steps to access multiple videos: generating an index for the plurality of videos based on one or more keyframes and one or more video features extracted from each respective video of the plurality of videos; processing user input to identify user intent; filtering the index to identify a number of candidate keyframes based on the user intent; and A method for generating a recommended video based on the plurality of candidate keyframes, comprising: selecting one or more additional frames based on processing the plurality of candidate keyframes using one or more clustering operations; and aggregating the plurality of candidate keyframes and the one or more additional frames; A procedure having A computer program comprising:

20. The steps to process user input and identify user intent are: converting the user input into text data; and using natural language processing (NLP) techniques to process the text data and predict the user intent; 20. The computer program of claim 19, comprising:

Images