GENERATION AND UTILIZATION OF DIGITAL MEDIA CLIPS BASED ON CONTEXTUAL METADATA FROM THE DIGITAL ENVIRONMENT - Patent application

Abstract

The present disclosure relates to a system, method, and non-transitory computer-readable medium for dynamically capturing, organizing, and utilizing digital media clips. For example, in one or more implementations, the disclosed system can capture and generate digital media clips of content items that include both content metadata of the content items and contextual metadata of contextual signals surrounding the content items. In addition, in some implementations, the disclosed system analyzes the contextual metadata to search, retrieve, discover, and organize new and existing digital media clips. Furthermore, in various implementations, the disclosed system facilitates the generation of digital media clip libraries, as well as the creation of digital media collections, where different types of digital media clips can be combined in a unified interactive graphical user interface.

Description

[Background technology]

[0001]

[0003] Recent years have seen significant advances in hardware and software platforms that improve the management of digital content. As web-based features such as messaging, content sharing, browsing, and search continue to increase, many existing web-based systems provide ways to organize digital content acquired online. For example, these existing systems provide organizational features such as bookmark managers and image collection boards.

[0002] Despite advances in the field of digital content management, many existing systems face several technical shortcomings that result in inaccurate, inefficient, and inflexible operation. For example, many existing systems inaccurately capture digital content items. For example, when a web page is bookmarked, some existing systems provide a bookmark manager that records the web page's URL as a list or file. As another example, when an image is detected as being online, various existing systems add the image, along with its web link, to an online collection or board of digital images. However, these existing systems often only collect surface information about the digital content itself, which limits the existing systems' ability to utilize and process digital content in meaningful ways, such as providing insights based on secondary information about the digital content item.

[0003] Additionally, many existing systems are inefficient with respect to collecting, managing, and utilizing digital content. In particular, various existing systems often organize collected digital content haphazardly and inefficiently. As a result, both storing and retrieving digital content is inefficient and costs additional computing resources to implement. In some instances, existing systems provide cumbersome navigation and menus for users who need to store or retrieve digital content. By requiring user interaction to navigate multiple and / or separate windows and menus, existing systems also consume excessive computing resources, such as real-time memory. Furthermore, many existing systems often fail to surface stored digital content in a meaningful way and frequently present duplicate and / or otherwise unnecessary digital content when attempting to retrieve the digital content. Additionally, many existing systems often display content inefficiently. For example, existing systems often require the use of multiple applications to access web-based stored digital content. Furthermore, users often must manually navigate through various cumbersome user interfaces to request access to stored digital content.

[0004] Furthermore, existing systems are often rigid and inflexible. As a first theme, existing systems are often unable to accommodate the increasing complexity and interaction of different web-based sources and / or content item types. For example, many existing systems are limited to connecting and storing content items of a given type or category. In particular, these existing systems are often constrained to access the capabilities of a single web source or a single content item type. Indeed, many existing systems include structures specifically designed to isolate each web source or content item type. Additionally, there are various types of content items that cannot be captured or collected by existing systems.

[0005] Furthermore, just as existing systems isolate individual web sources for users, existing systems often isolate individual users from each other with respect to collecting digital content. While existing systems may allow users to access and communicate with particular web sources (e.g., online email or messaging systems), existing browsing systems are limited in allowing user collaboration across other web sources.

[0006] Additionally, various existing systems provide inadequate display and arrangement of content items. Many existing systems often cannot display different types of content items within the same user interface. Furthermore, many existing systems are inflexible in that they lack additional tools, capabilities, and operations for accurately and dynamically capturing and organizing content items without requiring substantial user input. In fact, this rigidity only exacerbates the accuracy and efficiency problems outlined above.

[0007] These present additional challenges and problems with existing web browsing systems. Summary of the Invention

[0008]

[0006] Embodiments of the present disclosure provide advantages and / or solve one or more of the aforementioned or other problems in the art with systems, non-transitory computer-readable media, and methods for dynamically capturing, organizing, and utilizing digital media clips. For example, in one or more implementations, the disclosed systems can capture and generate digital media clips of content items that include both content metadata for the content items and contextual metadata for contextual signals surrounding the content items. Additionally, in some implementations, the disclosed systems analyze the contextual metadata to search, retrieve, discover, and organize new and existing digital media clips. Furthermore, in various implementations, the disclosed systems facilitate the creation of digital media clip libraries and digital media collections, and can combine different types of digital media clips in a unified, interactive graphical user interface.

[0009] To illustrate, in one or more implementations, the disclosed system detects user interaction with a content item and, in response, extracts the content item and content metadata for the content item. Additionally, the disclosed system can generate contextual metadata for the content item based on contextual signals from the digital environment in which the content item is located. The disclosed system can also generate a digital media clip that includes the content item, the content metadata, and the contextual metadata. Furthermore, the disclosed system can identify the content item based on the contextual metadata from the digital media clip. Additionally, in some examples, the disclosed system can display the content item along with the content item metadata within an interactive graphical user interface.

[0010] Additionally, the media clip system can provide recommendations for adding content items to a media clip library of a user account. To illustrate, in various implementations, the media clip system identifies items that are not part of a media clip library of a user account. Additionally, the content management system can determine a correlation score for each content item by comparing it with contextual metadata from media clips in the media clip library. Based on the correlation score, the content management system can provide recommendations for adding content items to a media clip library of a user account.

[0011] Additionally, the disclosed system can generate and provide various versions of a media clip collection. For example, in one or more implementations, the disclosed system provides one or more control lenses for the media clip collection, thereby modifying the appearance of or access to the media clip collection when viewed by others. In some implementations, the disclosed system provides various layout arrangements for viewing, editing, and / or presenting media clips, including flexible, free-form layout configurations. Additionally, in various implementations, the disclosed system provides stacking of digital media clips to combine the digital media clips together when viewed within a media clip collection.

[0012] Additional features and advantages of one or more embodiments of the present disclosure are outlined in the description below. [Brief explanation of the drawings]

[0013] The detailed description, as briefly described below, provides one or more embodiments with additional specificity and detail through the use of the accompanying drawings.

[0014] [Figure 1] FIG. 1 shows a diagram of a computing system environment that includes a digital media clip system according to one or more embodiments.

[0015] [Figure 2] FIG. 2 illustrates a schematic diagram of capturing, generating, and utilizing digital media clips, according to one or more embodiments.

[0016] [Figure 3] FIG. 3 shows a diagram of extracting content metadata for a content item, generating contextual metadata, and generating a digital media clip, according to one or more embodiments.

[0017] [Figure 4A] , [Figure 4B] , [Figure 4C] 4A-4C illustrate generating digital media clips based on different content item types according to one or more embodiments.

[0018] [Figure 5A] , [Figure 5B] 5A-5B illustrate various interactive interfaces for capturing digital media clips, according to one or more embodiments.

[0019] [Figure 6A] , [Figure 6B] 6A-6B illustrate various interactive interfaces for viewing and utilizing digital media clips, according to one or more embodiments.

[0020] [Figure 7] FIG. 7 illustrates searching for digital media clips based on contextual metadata, according to one or more embodiments.

[0021] [Figure 8] FIG. 8 illustrates capturing and associating additional content items with contextual metadata of a digital media clip, according to one or more embodiments.

[0022] [Figure 9] FIG. 9 shows a flowchart of a series of operations for generating a digital media clip in accordance with one or more embodiments.

[0023] [Figure 10] FIG. 10 shows a schematic diagram of recommending content items for addition to a media clip library based on contextual metadata, according to one or more embodiments.

[0024] [Figure 11] FIG. 11 illustrates suggesting adding a content item to a user account's media clip library based on contextual metadata, according to one or more embodiments.

[0025] [Figure 12] FIG. 12 illustrates suggesting adding a content item from a media clip library of a second user account to a media clip library of a first user account, according to one or more embodiments.

[0026] [Figure 13] FIG. 13 illustrates generating personalized search results for content items based on implicit user preferences, according to one or more embodiments.

[0027] [Figure 14] FIG. 14 illustrates tuning and utilization of a media clip classification machine learning model for recommending content items, according to one or more embodiments.

[0028] [Figure 15]FIG. 15 illustrates a flowchart of a series of operations for recommending content items to add to a media clip library based on contextual metadata, in accordance with one or more embodiments.

[0029] [Figure 16] FIG. 16 shows a schematic diagram of generating a media clip collection using various control lenses and layout configurations in accordance with one or more embodiments.

[0030] [Figure 17A] , [Figure 17B] , [Figure 17C] , [Figure 17D] , [Figure 17E] , [Figure 17F] 17A-17F illustrate creating a media clip collection according to one or more embodiments.

[0031] [Figure 18A] , [Figure 18B] 18A-18B illustrate utilizing stacked digital media clips according to one or more embodiments.

[0032] [Figure 19A] , [Figure 19B] , [Figure 19C] 19A-19C illustrate utilizing various layout arrangements of a media clip collection in accordance with one or more embodiments.

[0033] [Figure 20A] , [Figure 20B] , [Figure 20C] 20A-20C illustrate the use of various control lenses according to one or more embodiments.

[0034] [Figure 21] FIG. 21 illustrates publishing and sharing a media clip collection according to one or more embodiments.

[0035] [Figure 22A] , [Figure 22B] 22A-22B illustrate adding attributed digital media clips and media clip collections to a media clip library according to one or more embodiments.

[0036] [Figure 23] FIG. 23 shows a flowchart of a series of operations for providing one or more control lenses for a media clip collection, according to one or more embodiments.

[0037] [Figure 24] FIG. 24 illustrates a block diagram of an exemplary computing device for implementing one or more embodiments.

[0038] [Figure 25] FIG. 25 illustrates a network environment for a content management system according to one or more embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0039] One or more embodiments of the present disclosure relate to a digital media clip system that dynamically captures, generates, organizes, and utilizes digital media clips. For example, the digital media clip system (or simply "media clip system") can generate digital media clips (or simply "media clips") for content items that include contextual metadata based on contextual signals identified when capturing the content items. Additionally, in some examples, the media clip system can analyze the contextual metadata to search, retrieve, discover, and organize new and existing digital media clips. Furthermore, in various examples, the media clip system can generate digital media collections that include various organizational schemes that can combine different types of digital media clips into a unified, interactive graphical user interface.

[0040] To illustrate, in some implementations, a media clip system detects a user interaction with a content item, such as detecting a request to "clip" the content item. In response, the media clip system extracts the content item and content metadata for the content item. Additionally, the media clip system can generate contextual metadata for the content item based on contextual signals from the digital environment in which the content item is located. The media clip system can also generate a digital media clip that includes the content item, the content metadata, and the contextual metadata. In additional implementations, the media clip system can identify the content item based on the contextual metadata from the digital media clip. Additionally, in some examples, the disclosed system can also display the content item along with its content item metadata within an interactive graphical user interface.

[0041] As described above, in various implementations, a media clip system detects requests to generate media clips from content items. For example, the media clip system provides an interactive interface to facilitate requests to capture content items and generate media clips from those content items. As provided below, the media clip system facilitates capturing media clips from content items across a wide range of content item types, applications, and digital environments.

[0042] In various implementations, in response to detecting a request to save a content item as a media clip, the media clip system extracts the content item from its source. As one example, the source of the content item is web-based, such as an image or text from a website, a social media post, or a video. As another example, the source of the content item is application-based, such as from a natively running application, such as a document, a game, a video chat, or a presentation. As a further example, the source of the content item is a screenshot or screen recording.

[0043] As described above, the media clip system can also capture content metadata for a content item. Examples of content metadata include the content item's name, size, file, type, attributes, tags, creation date, or permissions. In some implementations, the type of content metadata available is based on the content item type of the content item. For example, an image has different content metadata than a document or web link. In an exemplary implementation, the type of content metadata available is based on the source of the content item. For example, an image extracted from a camera may include additional content metadata beyond the content metadata of an image screenshot. Thus, the media clip system can determine, detect, and collect content metadata for a content item based on the content item type, source, and / or other factors.

[0044] Also, as described above, the media clip system can generate contextual metadata for a content item. For example, in one or more implementations, the media clip system identifies one or more digital environments associated with a client device that provides the content item. Examples of digital environments include a web-based environment, an application environment, and / or an operating system (OS) environment. In various implementations, each of these digital environments provides one or more signals (i.e., context signals) that the media clip system can collect and analyze to discover additional context for the content item at the time of capture.

[0045] In various implementations, the media clip system generates contextual metadata for the content item based on these contextual signals. For example, when an image (e.g., a content item) is captured within a web page browser, the media clip system can detect web-based environmental signals (e.g., the URL of the website where the image was captured, the source of the image, and / or other tabs that were recently opened or accessed), application-based signals (e.g., open or active applications or processes), and OS signals (e.g., capture date and time, OS system details, network activity, etc.). In some implementations, the media clip system stores one or more signals as contextual metadata for the content item. In various implementations, the media clip system analyzes one or more signals and stores the results as contextual metadata.

[0046] As described above, the media clip system can generate media clips. For example, in various implementations, the media clip system generates media clips by combining a content item with its content metadata and contextual metadata. For example, in some implementations, the media clip system stores the content metadata and / or contextual metadata in a digital media clip metadata database (or simply a "metadata database") and generates media clips by associating the content item with the content item metadata (i.e., the content metadata and the contextual metadata) in the metadata database. In alternative implementations, the media clip system generates a new file type that includes the content item along with its content item metadata.

[0047] In various implementations, a media clip system generates a digital media clip collection that includes a plurality of media clips. In one or more implementations, the media clip system facilitates identifying, displaying, and searching for existing media clips based on content items and / or metadata (including contextual metadata). In various implementations, the media clip system provides an interactive interface for displaying, navigating, searching, and otherwise utilizing the media clip collection.

[0048] Additionally, the media clip system can provide recommendations for adding content items to a media clip collection and / or a media clip library. To illustrate, in various implementations, the media clip system identifies one or more content items that are not associated with a user's content management system account, such as content items in a web browser. For these content items, the media clip system can determine a correlation score by comparing the content items with contextual metadata from media clips in the media clip library or media clip collection. Based on the correlation score, the media clip system can provide recommendations for adding the content items to the media clip collection or media clip collection. In particular, the media clip system can generate media clips for the recommended content items and add the media clips to a media clip library associated with the user account.

[0049] The media clip system can also suggest adding content items to a user account's media clip library based on media clips that belong to the media clip libraries of other user accounts. For example, in various implementations, the media clip system may determine that two user accounts contain the same content item in their respective media clip libraries. Thus, the media clip system may analyze media clips from one user account's media clip library and recommend adding one or more of these media clips to the other user's media clip library. In various implementations, the media clip system makes recommendations based on comparing contextual metadata.

[0050] In some implementations, the media clip system improves search functionality for identifying content items stored in digital media clips in a media clip library of a user account. To illustrate, in various implementations, the media clip system identifies a set of content items based on a search query, where the search query may include detecting a text string or a selection of a target media clip. Additionally, the media clip system identifies implicit user preferences based on contextual metadata of the media clips in the media clip library. Additionally, the media clip system generates a subset of content items by filtering the set of content items based on the implicit user preferences and presents the subset of content items in response to a search request.

[0051] The media clip system can also facilitate relevance searches for new content items. For example, in one or more implementations, the media clip system receives a search query and a relevance level indication, which may range from highly correlated to unrelated or uncorrelated. Furthermore, in some implementations, the media clip system tunes parameters of a media clip classification machine learning model based on the relevance level. Furthermore, the media clip system utilizes the tuned media clip classification machine learning model and the search query to identify content items that match the search query to the extent indicated by the relevance level indication, as described further below.

[0052] In various implementations, the media clip system facilitates generating a media clip collection from one or more digital media clips in a media clip library of a user account in the content management system. For example, the media clip system provides an interactive interface that displays digital media clips belonging to a user account along with tools for creating a media clip collection from the digital media clips. In various implementations, the content management system facilitates adding digital media clips and content items to the media clip collection from other user accounts in the content management system and / or from external sources. Additionally, in some examples, the content management system determines and displays metadata for the media clip collection when presenting the media clip collection.

[0053] In some implementations, in connection with creating a media clip collection, the media clip system can provide suggestions, recommendations, and insights for media clips added to the media clip collection. For example, the content management system suggests similar media clips from the user account's media clip library or from other user account's media clip libraries. In some examples, the content management system provides insights such as the background of the digital media clip and / or additional content item metadata discovered about the media clip.

[0054] In various implementations, the media clip system facilitates creating and utilizing various control lenses with a media clip collection. For example, in one or more implementations, the content management system adds one or more control lenses that change the appearance and / or access to a media clip collection without modifying the digital media clips in the media clip collection. Furthermore, the content management system can apply different control lenses to different viewers. In this manner, the content management system can change how a media clip collection appears for one viewer and how the media clip collection can be accessed for another viewer.

[0055] Additionally, in some implementations, the content management system generates stacked digital media clips. For example, stacked digital media clips may include multiple digital media clips combined or spliced ​​in a defined presentation order. Indeed, when two or more digital media clips are stacked, the content management system displays each of the digital media clips at least partially together when presented in a media clip collection. In this way, when one of the digital media clips in the stack is selected for display, each of the digital media clips in the stack is displayed simultaneously.

[0056] In some implementations, the content management system generates and provides several layout configurations. For example, the content management system can provide a variety of layout configurations ranging from rigid layouts to flexible configurations. Examples of layout configurations include grid layouts, gallery layouts, timeline layouts, and freeform layouts (e.g., canvas layouts), among other layout configurations.

[0057] Also, as described above, the digital media clip system can offer several advantages over existing systems, particularly with respect to accuracy, efficiency, and flexibility of implementing a computing device. Illustratively, the media clip system improves accuracy over existing systems by generating digital media clips that provide a more complete representation of a content item. In effect, the media clip system generates and utilizes media clips that pair content items with contextual metadata (in addition to content metadata). By adding contextual information (e.g., web-based environmental signals, application-based environmental signals, and / or OS-based environmental signals) to content items within media clips, the media clip system facilitates improved organization and search accuracy and other functions between content items (i.e., the media clip system).

[0058] In addition to improved accuracy, the digital media clip system can also improve efficiency compared to existing systems. For example, the media clip system improves efficiency related to the collection, management, and utilization of media clips and media clip collections. In particular, in various implementations, the media clip system better organizes collections of media clips based on their contextual metadata (e.g., intelligent organization of media clips within a collection), which can often provide better relationship signals between media clips than other information. As a result, the media clip system provides improved storage and retrieval of media clips, which further reduces computing costs and reduces resource usage. Indeed, in various implementations, the media clip system prevents frequent duplication of content items and reduces resource usage compared to existing systems.

[0059] In various implementations, the media clip system improves efficiency by providing an improved interactive interface (i.e., an interactive graphical user interface). For example, the media clip system can provide various streamlined interactive interfaces that enable more efficient capture, search, and organization functions than existing systems. As a result, the media clip system reduces the number of navigation steps previously required to store, search, utilize, and / or otherwise organize media clips.

[0060] Additionally, the media clip system can provide various interactive interfaces to assist in capturing content items as media clips. In some implementations, the media clip system provides an interactive interface for searching for new or existing content items based on their contextual metadata. In one or more implementations, the media clip system provides an interactive interface for creating various media clip collections and / or modifying the organization or configuration of media clip collections. Furthermore, the interactive interfaces enable access to media clips without switching applications or navigating between different interfaces of the application. Indeed, in one or more implementations, the media clip system provides a floating interactive interface or browser extension that enables quick and efficient searching for media clips as well as the ability to drag media clips into an application without leaving the application.

[0061] As described above, the digital media clip system may also provide increased flexibility compared to existing systems. For example, the media clip system may generate, organize, and / or utilize media clips for content items across a wide range of content item types. For example, in various implementations, the media clip system facilitates dynamically sized media clip collections that display media clips from a wide variety of content item types. In contrast, most existing systems are limited to a single content item type or grouping of content item types.

[0062] Additionally, the media clip system facilitates using and synchronizing media clips, media clip collections, and media clip libraries across different client device platforms and within web-based environments. Similarly, the media clip system can efficiently scale up based on the number of media clips, media clip collections, user accounts, and / or other growth factors.

[0063] As indicated by the above discussion, this disclosure utilizes various terms to describe the features and advantages of the digital media clip system. Further details regarding the meaning of some of these terms are now provided. For example, as used herein, the term "content item" refers to a discrete digital data representation of a document, file, image, or another data structure having digital content. In particular, content items can include, but are not limited to, electronic documents (e.g., text files, spreadsheets, PDFs, web pages, presentations, lists, tables, forms, workspaces, templates, playlists, posts, calendar items, mailed messages, etc.), digital images, digital video cameras, and / or electronic communications.

[0064] As used herein, the term “content metadata” refers to attributes, characteristics, and / or descriptions of a content item. In some embodiments, content metadata refers to data associated with a content item at the time of capture of the content item. For example, before a user captures a content item, the content item may be associated with metadata (e.g., tags, links, or other forms of data) that describe aspects, attributes, characteristics, and / or descriptions of the content item. Examples of content metadata include the content item's name, size, file, type, attributes, tags, creation date, or permissions. Content metadata may also vary based on content item type. For example, an image content item type may include content metadata such as resolution, image capture date, image capture location, pixel density, and file size, while a text document may have other content metadata such as author, version data, keyword data, and page count. In one or more implementations, a media clip system collects content metadata from content item sources. In some implementations, a digital content management system determines, derives, or otherwise determines the content metadata of a content item.

[0065] As used herein, the term “contextual metadata” refers to contextual information associated with a content item. In some embodiments, contextual metadata is data generated when a media clip system captures a content item, e.g., contextual metadata refers to data that is not pre-associated with a content item before capture. In some implementations, contextual metadata includes metadata that is not pre-specified and not previously associated with a content item; thus, in some examples, contextual metadata includes data that requires post-capture processing to derive additional attributes, aspects, and properties of a content item that are not included in the content item metadata. For example, a media clip system can process a content item with processes such as natural language processing, image processing, or other post-processing techniques to derive contextual metadata. Furthermore, content items are often located within one or more digital environments. Examples of digital environments include web-based environments, application-based environments, or operating system environments. Each digital environment can generate context signals (or simply “signals”) that reveal or indicate contextual information such as capture time, capture location, current applications open, hidden windows, application version information, OS version information, etc. Furthermore, the term “content item metadata” refers to the combination of a content item’s content metadata and contextual metadata.

[0066] The terms "media clip library" and "digital media clip library" refer to digital media clips associated with a user account. For example, each time a user account adds a digital media clip, the media clip system associates it with the user account's media clip library. Furthermore, the terms "media clip collection" and "digital media clip collection" refer to a group of digital media clips associated with one or more user accounts. For example, the media clip system generates a media clip collection from one or more digital media clips in the user account's media clip library. Furthermore, when a digital media clip from a first user account is added to a second user account's media clip collection, the media clip system can add the digital media clip to the second user account's media clip collection as well as the second user account's media clip library.

[0067] Additionally, as used herein, the term "control lens" refers to a filter, layer, or element that modifies access to and / or presentation of one or more digital media clips. As one example, a control lens added to a media clip collection changes the appearance of the media clips in the media clip collection. As another example, another control lens added to a media clip collection restricts, limits, or modifies how the media clips in the media clip collection are interacted with or viewed. In various implementations, the media clip system adds metadata to the media clip collection that provides instructions for the control lens.

[0068] As used herein, the term "machine learning model" refers to a computer model or computer representation that can be tuned (e.g., trained) based on inputs to approximate an unknown function. For example, machine learning models can include, but are not limited to, transformer models, sequence-to-sequence models, natural language processing (NLP) models, neural networks (e.g., convolutional neural networks or deep learning models), decision trees (e.g., gradient-boosted decision trees), linear regression models, logistic regression models, random forest models, clustering models, association rule learning, inductive logic programming, support vector learning, Bayesian networks, regression-based models, principal component analysis, or combinations thereof.

[0069] Furthermore, as used herein, the term "neural network" refers to a machine learning system or algorithm that can be tuned (e.g., trained) based on training inputs to estimate unknown functions. In particular, a neural network can include multiple interconnected artificial neurons that send data to other artificial neurons, which generate outputs based on one or more inputs. More specifically, the multiple interconnected neurons can learn to estimate complex elements by utilizing prior estimates and other training data. For example, a neural network may include a deep neural network, a convolutional neural network ("CNN"), a fully convolutional neural network ("FCN"), or a recurrent neural network ("RNN"). In other words, a neural network is a system or algorithm that implements deep learning techniques that utilize a set of learning parameters arranged in layers according to a specific architecture to attempt to model high-level abstractions in data.

[0070] Further details regarding the digital media clip system will now be provided with reference to the drawings. For example, Figure 1 shows a schematic diagram of a digital media system environment 100 (or simply "environment 100") for implementing a digital media clip system 106 according to one or more embodiments. As shown in Figure 1, environment 100 includes a client device 102 and a server device 108 connected via a network 116. Further details regarding these computing devices and the network are provided below in connection with Figures 24-25.

[0071] As described above, environment 100 includes server device 108. In some embodiments, server device 108 comprises a content server and / or a data collection server. As shown, server device 108 includes content management system 110 having digital media clip server system 112. Content management system 110 can organize, manage, and / or perform tasks related to user accounts, cloud storage, file synchronization, data security / encryption, smart workspaces, etc. In one or more implementations, content management system 110 facilitates the creation, management, storage, and / or utilization of media clips associated with users' accounts (i.e., "user accounts") within content management system 110.

[0072] In at least one embodiment, content management system 110 organizes digital content (e.g., digital media clips and other content items such as user-generated documents and images) and stores changes made to the digital content in response to various user activities. For example, in response to a user generating a new content item using client device 102, content management system 110 on server device 108 detects (via network 116) the content item, stores the content item, and / or distributes the content item (or a placeholder for the content item) to other client devices (or shared user accounts) associated with the user's account. Additional details regarding content management system 110 are provided below in conjunction with FIG. 25.

[0073] As shown, the content management system 110 includes a digital media clip server system 112. In one or more implementations, the digital media clip server system 112 supports and / or operates in conjunction with the media clip system 106 on the client device 102. For example, the digital media clip server system 112 provides instructions to the content management system application 104 on the client device 102 to implement one or more functions of the media clip system 106.

[0074] As shown in FIG. 1, environment 100 includes client device 102. Client device 102 may be one of a variety of computing devices, including a smartphone, a tablet, a mobile terminal, a smart TV, a desktop computer, a laptop computer, a virtual reality device, an augmented reality device, and / or another computing device, as described in FIG. 25. For example, client device 102 may be operated by a user (e.g., via content management system application 104) to perform various functions, such as, but not limited to, creating, receiving, sorting, browsing, searching, identifying, modifying, and / or sending media clips, configuring content management system user account or application settings, and / or electronically communicating with other user accounts in the content management system. While FIG. 1 shows a single client device, in some embodiments, environment 100 may include any number of similar client devices.

[0075] To access the functionality of the content management system 110 (and the media clip system 106), a user can interact with a content management system application 104 via a client device 102. The content management system application 104 may include one or more software applications installed on the client device 102. In some implementations, the content management system application 104 may include one or more software applications downloaded and installed on the client device 102 to include an implementation of the media clip system 106. In some embodiments, the content management system application 104 is hosted on the server device 108 and accessed by the client device 102 via a web browser and / or another online platform. Additionally, the content management system application 104 may include functionality for accessing or modifying file structures stored locally on the client device 102 and / or hosted on the server device 108.

[0076] As mentioned above, in some embodiments, the client device 102 includes a media clip system 106 and a digital media clip metadata database 114 (or simply “metadata database 114”). In particular, as shown in FIG. 1 , the client device 102 implements the media clip system 106 and the digital media clip metadata database 114 through a content management system application 104. In one or more implementations, the media clip system 106 generates, captures, produces, organizes, and utilizes digital media clips. In some implementations, the media clip system 106 searches for and recommends media clips based on the media clips' contextual metadata. In various implementations, the media clip system 106 provides clips within a collection and / or digital canvas for viewing, editing, and / or performing other functions, as provided above. Additionally, in certain implementations, the media clip system 106 maintains some or all of the media clips in the digital media clip metadata database 114. For example, the media clip system 106 stores content metadata and / or contextual metadata on the digital media clip metadata database 114. In some implementations, the digital media clip metadata database 114 stores a collection of media clips. Further details regarding the media clip system 106 and the digital media clip metadata database 114 are provided below in subsequent figures.

[0077] In some implementations, the media clip system 106 and / or the digital media clip metadata database 114 are located on the client device 102, external to the content management system application 104. While FIG. 1 illustrates the media clip system 106 implemented by particular components and / or devices within the client device 102, in some embodiments, the media clip system 106 is implemented, in whole or in part, by other computing devices and / or components within the environment 100. For example, as shown, in some implementations, the media clip system 106 is implemented on the server device 108 within the content management system 110. More specifically, in some embodiments, some or all of the media clip system 106 is implemented by the server device 108 and accessed by the client device 102 via the content management system application 104, a web browser, and / or other online platforms (as described above). Creating and using digital media clips containing contextual metadata

[0078] As described above, in various implementations, the media clip system 106 generates, organizes, and / or utilizes media clips. To illustrate, Figure 2 shows a schematic diagram of generating a media clip for a content item and then identifying the content item based on corresponding contextual metadata. In particular, Figure 2 includes a series of operations 200 performed by the media clip system 106 to generate a media clip.

[0079] 2, the series of operations 200 includes an operation 202 of detecting a request to clip a content item in a digital media clip library. For example, in some implementations, the media clip system 106 detects a user interaction with an external content item. For example, the media clip system 106 detects a request to "clip" (e.g., save) the content item and add it to a media clip collection (e.g., a set or grouping of one or more media clips associated with a user account on the content management system) and / or to the user account's media clip library. Further details regarding detecting a request to clip a content item are provided below in connection with FIGS. 4A-4C and 5A-5B.

[0080] The series of operations 200 also includes an operation 204 for extracting content items and content metadata. For example, in some embodiments, the media clip system 106 saves, copies, records, captures, and / or screenshots the content items. In various implementations, the media clip system 106 determines how to capture the content items based on the content item type. Additionally, in various implementations, the media clip system 106 extracts content metadata (e.g., attributes, characteristics, and descriptions about the content items) of the content items. Further details regarding the extraction of content items and content metadata are provided below in connection with FIGS. 3 and 4A-4C.

[0081] 2, the series of operations 200 includes an operation 206 of generating contextual metadata based on the content metadata. For example, in one or more implementations, the media clip system 106 utilizes the content metadata to discover information about the environment in which the content item resides. In some implementations, the operation includes generating contextual metadata for the content item based on context signals from one or more digital environments in which the content item is located (e.g., with or without content metadata). Further details regarding the generation of contextual metadata are provided below in connection with FIGS. 3, 4A-4C, and 8.

[0082] Further, the series of operations 200 includes an operation 208 that generates a digital media clip by combining the content items, the content metadata, and the contextual metadata. For example, in various implementations, the media clip system 106 groups the captured content items with their corresponding content metadata and the generated contextual metadata. In some implementations, the media clip system 106 associates the content items, which may be stored in a file system, with the content metadata and contextual metadata that generate a media clip, which may be stored in a metadata database and represented in a media clip collection. Further details regarding the generation of media clips are provided below in connection with FIGS. 3 and 6A-6B below.

[0083] 2 , the series of operations 200 includes an operation 210 of identifying a content item based on contextual metadata. For example, in one or more implementations, the media clip system 106 detects user input within an interactive interface to search for content items based on contextual metadata. For example, the media clip system 106 receives a search query for "Clipped last week" or another environmental signal along with the search term. In response, the media clip system 106 can identify matching contextual metadata. Further, the media clip system 106 returns content items that correspond to the identified contextual metadata. Further details regarding searching for and identifying content items based on contextual metadata are provided below in connection with FIG. 7 .

[0084] Referring now to the next figure, Figure 3 illustrates a diagram of extracting content metadata for a content item, generating contextual metadata, and generating a digital media clip, according to one or more embodiments. In particular, Figure 3 expands upon acts 208-208 from the sequence of acts 200 shown in Figure 2. In fact, each of the acts in Figure 3 includes multiple sub-acts.

[0085] 3 includes operation 204 for extracting content items and content metadata. As shown, operation 204 includes sub-operation 310 for identifying a content item type. For example, in various implementations, the media clip system 106 determines the data structure, media type, protocol, and / or file type of the content item. For example, the media clip system 106 determines whether the content item is an image, a text string, a video, an animation, a social media post, a token (e.g., a non-fungible token or NFT), an XML file, a document, an application file, a screenshot, an audio segment, or another content item type.

[0086] In various implementations, the media clip system 106 detects the content item type from the content item's file extension. For example, if the content item is located natively on the client device, the media clip system 106 identifies the file extension type. If the content item is web-based, the media clip system 106 can identify the file extension type from a network source. In some implementations, such as when the file type is unknown, the media clip system 106 utilizes trained machine learning models and / or neural networks to analyze images, text, and / or audio from the content item to determine the content item type.

[0087] As shown, operation 204 includes sub-operation 312 of copying a content item based on the content item type. For example, the media clip system 106 may utilize a set of heuristics to determine how to capture a content item. For example, the media clip system 106 may utilize a first set of rules or actions for capturing images and a second set of rules or actions for capturing text. In some implementations, the media clip system 106 enables a new set of capture tools based on the content item type. To illustrate, for live video, the media clip system 106 may provide a capture tool to save the last 30 seconds (or another length) of the video (in some examples, the entire live video user interface), and the media clip system 106 may capture all or a selected portion of a pre-recorded video. In fact, the media clip system 106 may continually add new heuristics to capture new content item types. In this manner, the media clip system 106 enables the capture of many types of content items that cannot be captured by existing systems.

[0088] Additionally, in some implementations, the media clip system 106 can capture content items based on their location. For example, for web-based content items (e.g., links, images, media, etc.), the media clip system 106 copies a link to the content item and / or downloads the content item from its network source. For locally stored or captured items, the media clip system 106 copies the content item locally or generates a placeholder (e.g., an alias or shortcut) for the content item.

[0089] In various implementations, the media clip system 106 stores the captured content items. For example, the media clip system 106 stores the captured content items in a file structure associated with a user account. In some implementations, the media clip system 106 stores the captured content items in a media graph (e.g., a data structure that is invisible to the client device user having a file structure). In some implementations, the media clip system 106 copies and / or synchronizes the captured content items to a network storage device and locally stores links to the remotely stored content items.

[0090] As shown, operation 204 includes a sub-operation 314 of the media clip system 106 identifying content metadata for the content item. For example, in one or more implementations, the media clip system 106 identifies a set of metadata attributes stored with the content item. In some examples, the media clip system 106 detects and captures a metadata file or data structure linked to the content item.

[0091] In some implementations, the media clip system 106 determines which attributes and / or characteristics of a content item (e.g., content metadata) to capture based on the type of content item. For example, each content item type can be associated with a list of content metadata for the corresponding content item. For example, for an image content item type, the media clip system 106 attempts to obtain image attributes such as image size, resolution, dimensions, color model, etc. Then, for a text content item type, the media clip system 106 attempts to obtain text attributes such as font type, font size, font style, etc.

[0092] In one or more implementations, the media clip system 106 stores the content metadata in a metadata database (i.e., a digital media clip metadata database). For example, the media clip system 106 maintains (e.g., locally or remotely) a metadata database that is indexed to content items and includes entries for content metadata corresponding to each content item. In some implementations, the metadata database may also include copies of the content items. In various implementations, the media clip system 106 stores the content metadata with the content items.

[0093] As shown, operation 204 includes a sub-operation 316 of the media clip system 106 analyzing the content item to determine additional content metadata. For example, in various implementations, the media clip system 106 utilizes an attribute detection model, an attribute detection machine learning model, and / or an attribute detection neural network to determine one or more attributes and / or characteristics of the content item. For example, for an image, the media clip system 106 can determine the color theme, color palette, or prominent colors found in the picture and save this information as content metadata. As another example, the media clip system 106 can detect text and / or objects in the image and include that information as content metadata.

[0094] As another example, the media clip system 106 can provide the text of a content item to an attribute detection model to discover characteristics and attributes about the text. For example, the attribute detection model may determine word patterns, word or sentence statistics, keywords, statistical phrases, etc. from analyzing the text content item. Indeed, for different content items and / or content item types, the attribute detection model may determine additional attributes and add this information to the content metadata of the content item.

[0095] 3 also includes an operation 206 of the media clip system 106 that generates contextual metadata based on the content metadata. As shown, operation 206 includes a sub-operation 320 that identifies a digital environment for the content metadata. For example, in connection with capturing a content item and / or the content metadata, the media clip system 106 can identify one or more digital environments in which the content item resides. As discussed above, examples of digital environments include a web-based environment, an application-based environment, and an OS environment. Often, a digital environment is located within another digital environment (e.g., an application-based environment is nested within an OS environment).

[0096] In some implementations, the media clip system 106 detects multiple digital environments of the same type. For example, while the media clip system 106 detects a single OS environment, the media clip system 106 also detects multiple application-based environments corresponding to multiple applications open and / or in use on the client device. Additionally, in an example implementation, the media clip system 106 detects that a content item is present within a web browser or web-based environment.

[0097] As shown, operation 206 includes sub-operation 322 of detecting signals (i.e., context signals) from the digital environments. For example, upon identifying various digital environments, media clip system 106 can analyze the digital environments to further identify context signals from each of the digital environments. As used herein, the term “environment signal” (or simply “signal”) refers to an action, process, attribute, or state of a digital environment. For example, a signal can indicate the digital environment version (e.g., OS version or web browser version), whether a given digital environment is active or implemented (e.g., in the case of an application-based environment), or the hardware profile of a client device. Additionally, a signal can indicate the current state of a digital environment, such as running processes, open applications, recently established network connections (e.g., recently loaded web pages), open web browser tabs, and recently accessed documents.

[0098] To further explain, in some implementations, suboperation 322 includes the media clip system 106 detecting an OS signal corresponding to the current date and time when the content item was captured. Additionally, the media clip system 106 can detect a web-based signal corresponding to one or more network sources being evaluated at approximately the same time as the web-based content item. Additionally, the media clip system 106 can detect an application-based signal corresponding to an open application. In this manner, the media clip system 106 can detect signals from one or more of the digital environments when clipping the content item.

[0099] As also shown, operation 206 includes sub-operation 324 of analyzing the signals to generate contextual metadata. By way of example, the media clip system 106 generates the contextual metadata by collecting, extracting, identifying, analyzing, and / or copying contextual signals, which enables the media clip system 106 to encapsulate the digital environment of the content item at the time it was clipped. In effect, the media clip system 106 detects, identifies, and / or generates the contextual metadata such that the media clip system 106 can predict (e.g., via a machine learning model trained on the environmental signals or another model) the current general purpose of the content (e.g., for which general purpose the content item is being viewed, utilized, and / or saved).

[0100] To illustrate, in some implementations, based on detecting signals from one or more application-based environments, the media clip system 106 analyzes the signals to determine currently open applications on the client device, whether the applications are visible or hidden, whether content items are located within the applications, the size and location of open application interfaces on the client device's virtual desktop, and / or whether the applications are currently being utilized by one or more users on the client device. In some implementations, the media clip system 106 analyzes signals from the application-based environments to determine whether applications are accessing files (e.g., files are open) and the subject of the open files.

[0101] In one or more implementations, based on detecting and analyzing signals from the web-based environment, the media clip system 106 determines network-based contextual information for a content item. For example, the media clip system 106 identifies links to various content sources accessed (e.g., requested by HTTP calls and / or downloaded) around the time the content item was accessed. In addition, the media clip system 106 identifies other elements within the vicinity of the content item (e.g., text, images, audio, video, their source links) and / or links associated with open (or saved) tabs and windows.

[0102] In an exemplary implementation, the media clip system 106 can determine that a content item that shares a boundary with a target content item is proximate to the target content item. In an exemplary implementation, the media clip system 106 determines that a target content item nested within a parent content item and / or all content items within the parent content item are proximate to the target content item. In one or more implementations, the media clip system 106 determines that content items within the same web page of the target content item are proximate to the target content item. In some implementations, all content items within a threshold amount (e.g., number of pixels, set distance, screen percentage, etc.) are proximate to the target content item.

[0103] Additionally, in various implementations, content management system 110 generates contextual metadata based on multiple environmental signals (i.e., contextual signals). For example, content management system 110 generates contextual metadata elements that combine one or more signals from two or more different digital environments. For example, content management system 110 combines one or more OS-based signals with one or more web-based signals and / or one or more application-based signals. In this manner, content management system 110 determines contextual metadata that distinguishes and captures the environment in which the content item resides, which content management system 110 can later use to search for media clips or provide media clip suggestions.

[0104] The media clip system 106 can store information from the signal as contextual metadata for the content item (e.g., in a metadata database). In some implementations, the media clip system 106 processes one or more environmental signals from the digital environment to generate the contextual data, as described above. For example, upon identifying a date and time signal from an OS environment or a web-based environment, the media clip system 106 generates a capture date and time as part of the contextual metadata for the content item. Similarly, the media clip system 106 stores information about the capture source and / or capture source of the content item based on analyzing the environmental signals, as described above.

[0105] 3 also includes an operation 208 of the media clip system 106 that generates a digital media clip by combining content items, content metadata, and contextual metadata. As shown, operation 208 includes a sub-operation 330 that stores the content metadata in a metadata database (i.e., a digital media clip metadata database). As provided above, in various implementations, the media clip system 106 maintains a metadata database that stores and / or links content metadata for content items.

[0106] As shown, operation 208 includes a sub-operation 332 of the media clip system 106 storing the contextual metadata in a metadata database. For example, the media clip system 106 also stores the contextual metadata in association with the content metadata for the content item in the metadata database. In some implementations, the media clip system 106 generates content item metadata for the content item by combining the content metadata and the contextual metadata (e.g., also known as content item metadata) in the metadata database.

[0107] As also shown, operation 208 includes a sub-operation 334 of the media clip system 106 generating a digital media clip by combining a content item with content item metadata (e.g., content metadata and contextual metadata). For example, in some implementations, the media clip system 106 generates a media clip that links a content item stored in a first location with its content item metadata (e.g., content metadata and contextual metadata) stored in a second location, such that the content item appears as an integrated file when selected. In alternative implementations, the media clip system 106 generates and / or stores a new file that combines the content item with its content item metadata.

[0108] As further shown, operation 208 includes a sub-operation 336 of the media clip system 106 adding the digital media clip to a digital media clip library. For example, in various implementations, the media clip system 106 generates a media clip and stores it in a media clip library of a user account. In some implementations, the media clip system 106 also adds linking the media clip to a media clip collection. Furthermore, in some examples, when displaying a media clip (e.g., by itself or within a media clip collection), the media clip system 106 displays the content item along with some or all of the content item metadata (e.g., accessed from a metadata database or another locating location that stores content item metadata for the content item). An example of displaying a media clip with several metadata elements is shown in FIG. 17C.

[0109] 4A-4C illustrate a graphical user interface (GUI) of a media clip system 106 for generating media clips. Specifically, FIGS. 4A-4C illustrate generating digital media clips based on different content item types, according to one or more embodiments. For context, FIGS. 4A-4C include a client device 400 having a graphical user interface 402 for displaying visual and graphical components and elements (e.g., a computer display). In some implementations, the client device 400 is a representation of the client device 102 introduced above in connection with FIG. 1.

[0110] 4A, the graphical user interface 402 includes a first application 404a that displays a first content item 410 (e.g., an image of a shirt). For example, the first application 404a is a web browser that displays images, text, audio, video, and other network-based digital content. As shown, the first application 404a includes various tabs and shows an open tab that includes the first content item 410 along with corresponding text and elements.

[0111] As also shown, the graphical user interface 402 includes a first menu 416 that includes selectable clipping elements 412 for generating a media clip from a content item. For example, in various implementations, upon detecting a selection of the first content item 410 (e.g., a right mouse click or equivalent), the client device and / or media clip system 106 displays the first menu 416 that includes the selectable clipping elements 412. As provided below, the media clip system 106 can provide the selectable clipping elements 412 (or equivalent options) via alternative menus or elements.

[0112] Based on detecting the selection of the selectable clipping element 412, the media clip system 106 can generate a first digital media clip 420 for the first content item 410. As shown, the media clip system 106 links, extracts, and / or copies the first content item 410 to the first digital media clip 420 (e.g., content item 422 represents a copy of the first content item 410). In some implementations, the media clip system 106 identifies content metadata 424 and determines contextual metadata 426 for the content item, as provided above.

[0113] To illustrate, with respect to content metadata 424, the media clip system 106 collects content metadata 424 associated with a first content item 410. For example, the media clip system 106 identifies image attributes and characteristics associated with the first content item 410. As shown, the content metadata 424 includes a list of exemplary content metadata elements. Notably, some of the content metadata elements are general to the content item (e.g., name, file type, size), while other content metadata elements are specific to the content item type (e.g., resolution, camera information).

[0114] With respect to the contextual metadata 426, the media clip system 106 generates the contextual metadata 426 from the digital environment around the first content item 410 and / or the content metadata 424. For example, as shown, the media clip system 106 detects browser signals (e.g., web-based signals), application signals (e.g., application-based signals), and OS signals. In particular, as shown, the media clip system 106 determines, from the browser signals, neighboring content, such as content or content items within a proximate distance to the first content item 410 and / or on the same web page as the first content item 410. For example, the media clip system 106 identifies, as elements of the contextual metadata 426, content item title, price, availability options (e.g., "add to cart"), product information, product text summary, other products near the first content item 410, etc. In some implementations, the neighboring content corresponds to content and content items in other open tabs within the first application 404a.

[0115] In some implementations, the browser signals provide information about a web page that is not directly tied to the first content item 410. For example, the browser signals indicate the creator of the website, whether the website belongs to an entity (e.g., a person or a company), when the website was last updated, the website's read time if text is present, embedded tags or search terms, etc. In effect, the browser signals can enable the media clip system 106 to generate a semantic understanding of the web page and / or adjacent web pages when the content item is clipped.

[0116] Additionally, the contextual metadata 426 indicates that the media clip system 106 determines the contextual metadata from the OS signals. For example, upon capturing the first content item 410, the content management system 110 identifies the server source as well as the date and time of capturing the content item. In some implementations, the media clip system 106 utilizes a content management system application to detect the OS signals at the time of content item capture (in conjunction with a web-based plug-in that captures browser signals).

[0117] Additionally, the media clip system 106 can perform content analysis to determine additional contextual metadata 426, as shown in FIG. 4A. In some implementations, the media clip system 106 analyzes a content item to determine one or more contextual metadata 426 elements. For example, the media clip system 106 utilizes a color detection model to determine the color profile of the first content item 410. The media clip system 106 can add this determined contextual metadata as a new element of the contextual metadata 426 or replace an existing content metadata element. In one or more embodiments, the media clip system 106 can use the determined color profile to generate color extraction tags, such as those shown in the contextual metadata illustrated in FIG. 4A. Further examples of contextual metadata 426 that may be generated from content item analysis include permissions, object tags, interest scores, and other data generated from the analysis of the content item.

[0118] As described above, in various implementations, the media clip system 106 generates the first digital media clip 420 by combining or otherwise associating the content item 422 with content metadata 424 and contextual metadata 426, as shown. In this manner, the first digital media clip 420 not only provides the content item 422 along with the content metadata 424 for the content item 422, but the first digital media clip 420 also provides the contextual metadata 426 regarding the digital environment in which the content item 422 was captured, as provided above.

[0119] 4B shows a graphical user interface 402 that includes a second application 404b that displays a second content item 432 (e.g., a highlighted portion of text from a larger section of text). In particular, the second application 404b is a document editor located on the client device 400 that displays a document 430 that includes the second content item 432. As also shown, the graphical user interface 402 includes a third application 404c and a fourth application 404d. For example, the third application 404c is an image viewer / editor application, and the fourth application 404d is a workbook application. Although not shown, the graphical user interface 402 may include additional applications, including copies of the same application (e.g., multiple document editor or image viewer applications).

[0120] 4B includes a content item element 438. In some examples, the content item element 438 selectively appears in association with the document 430 and / or the second application 404b. In one or more implementations, the content item element 438 is part of an application executing on the client device 400, such as an application associated with a content management system.

[0121] As shown, the content management system 110 can detect a selection of the content item element 438, such as a click, hover, gesture, or another action. Upon detecting the selection of the content item element 438, the graphical user interface 402 can update to show a second menu 440 including selectable clipping elements 442 for generating a media clip from the content item. Furthermore, based on detecting a further selection of the selectable clipping element 442, the media clip system 106 can generate a second digital media clip 450 of the second content item 432.

[0122] To illustrate, the media clip system 106 copies (or otherwise extracts) the second content item 432 as content item 452. As shown, the media clip system 106 copies the highlighted text in the document 430 as content item 452 of the second digital media clip 450. Additionally, the media clip system 106 may also identify and / or generate content metadata 454 and contextual metadata 456, as described above. Specifically, the content metadata 454 may include general metadata attributes that correspond to multiple types of content items and more specific metadata attributes that correspond to the content item type of the content item 452. For example, the content metadata 454 associated with the selected text may include a name, a media type, and a source (e.g., a URL or file location), as shown in FIG. 4B .

[0123] In one or more embodiments of the media clip system 106, document 430 is a web page in a web browser, and document 432 is a web document, i.e., a web page. In such an example, the media clip system 106 may clip a portion of the web page (as shown in FIG. 4A ), or the media clip system 106 may clip an entire web page or website. In other examples, the media clip system 106 may allow a user to select an image or other content element within a web page. The features and functionality described herein with respect to selected text may similarly apply when the media clip system 106 saves a web page, website, or other elements within a website.

[0124] As described above, in various implementations, the media clip system 106 generates contextual metadata 456. As shown in FIG. 4B , the media clip system 106 detects application signals (e.g., application-based signals) and OS signals. Specifically, as shown, the media clip system 106 determines adjacent content from the application signals, such as elements and / or content items within the vicinity of the second content item 432. For example, the media clip system 106 adds the title of the document 430 and text surrounding the second content item 432 as contextual metadata 456, thereby allowing the second content item 432 to be considered in its intended context.

[0125] Additionally, the media clip system 106 analyzes the application signals to determine open documents and / or active applications. For example, in various implementations, the media clip system 106 detects whether the document 430 is connected to other open documents (e.g., tabs) within the second application 404b. Additionally, in some implementations, the media clip system 106 identifies other active applications (e.g., applications that are open, visible, and / or active at or around the time the second digital media clip 450 is requested), such as the third application 404c and the second application 404b. As described above, in some implementations, the media clip system 106 can utilize the application signals to determine the user's further actions and / or intentions regarding the second content item 432 and / or other potential media clips.

[0126] Further, in one or more embodiments, the media clip system 106 can generate content metadata 454 based on analyzing the captured text. For example, as shown in FIG. 4B, the media clip system 106 can analyze the text to determine authorship, extract keywords, and generate keyword tags. Additionally, as further shown in FIG. 4A, the media clip system 106 can analyze the text (e.g., using ML or classification models) to determine a sentiment score (e.g., positive, negative, neutral) and / or aesthetic classification (e.g., format, font, art style, etc.) associated with the text.

[0127] 4C shows the graphical user interface 402 including a fifth application 404e that displays a third content item 460 (e.g., a website including images and text). For example, the fifth application 404e is a web browser located on the client device 400 that displays a web page. In an alternative implementation, the fifth application 404e is another application type, such as a document editor, an image viewer, a video player, or the like.

[0128] As shown, the media clip system 106 facilitates generating a digital media clip 470 based on capturing a screenshot 462 of a third content item 460. For example, in various implementations, the media clip system 106 provides tools for capturing a screenshot image and generating a media clip from the image. Indeed, in one or more implementations, the media clip system 106 provides a menu with selectable clipping elements for capturing and storing the screenshot. In an exemplary implementation, upon detecting a screenshot, the media clip system 106 can prompt the user to generate a media clip from the screenshot.

[0129] As shown, the media clip system 106 generates a digital media clip 470 having a content item 472, content metadata 474, and contextual metadata 476. In various implementations, the media clip system 106 identifies and / or collects elements of the content metadata 474, as described above. In some implementations, the media clip system 106 analyzes the content item 472 to determine missing and / or additional content metadata. For example, screenshots of a content item are often newly created images and therefore may not have all of the content metadata 474 present. In these examples, the media clip system 106 can analyze the content item to identify additional corresponding content metadata 474, as provided above. In some implementations, the media clip system 106 attempts to complete the list of content metadata elements with initial null values ​​when the content item is a screenshot.

[0130] In some implementations, when capturing a screenshot, the media clip system 106 can identify elements of the content metadata 474 based on analyzing the corresponding content item identified in the screenshot. For example, in one or more implementations, the media clip system 106 identifies an image in the screenshot and locates the corresponding image in the graphical user interface 402 of the client device 400. The media clip system 106 then identifies and adds one or more content metadata elements to the content metadata 474 of the digital media clip 470. Similarly, the media clip system 106 can analyze the captured text and / or corresponding text to identify additional corresponding content metadata 474 for the content item 472.

[0131] As described above, the media clip system 106 can generate the contextual metadata 476. For example, the media clip system 106 identifies and analyzes contextual signals of one or more digital environments on the client device 400 to generate the contextual metadata 476 for the content item 472. For example, as shown, the media clip system 106 determines the content seen in the screenshot as well as adjacent content items to open applications.

[0132] Indeed, as shown in Figures 4A-4C, the media clip system 106 can generate media clips for a wide range of content item types. Additionally, the media clip system 106 can also identify and / or generate content and contextual metadata for captured content items. In some instances, the media clip system 106 identifies and / or determines different content metadata elements based on the content item type and the detected environmental signals. Additionally, the media clip system 106 can follow a framework for generating media clips that includes adding content and contextual metadata based on the content item type for content item types not shown in Figures 4A-4C.

[0133] Referring to the next set of figures, FIGS. 5A-5B illustrate various interactive interfaces for capturing digital media clips in accordance with one or more embodiments. To illustrate, FIG. 5A includes a content item 502, a first interactive interface 504, and a second interactive interface 506. Upon detecting an interaction with the content item 502, such as on a client device, the media clip system 106 may display the first interactive interface 504, the second interactive interface 506, or another interactive interface. In some implementations, the media clip system 106 may display the first interactive interface 504 if the client device is a mobile terminal or has a smaller, compact display (e.g., a smartphone or mini mobile terminal) and display the second interactive interface 506 for client devices with larger displays (e.g., a laptop or desktop).

[0134] To illustrate, in one or more implementations, the client device displays a first interactive interface 504. As shown, the first interactive interface 504 can be a menu that includes a first media clip preview 508a of the content item 502 and a first selectable clipping element 509a. In some implementations, the first media clip preview 508a includes a limited preview to accommodate smaller screens. For example, the first media clip preview 508a includes a small image of the content item 502, a title, and a content item type (e.g., “Image”).

[0135] In various implementations, the client device displays a second interactive interface 506. As shown, the second interactive interface 506 includes a second media clip preview 508b of the content item 502 and a second selectable clipping element 509b. As also shown, the second media clip preview 508b includes an image of the content item 502 along with corresponding information. In one or more implementations, the media clip system 106 adds content metadata to the second media clip preview 508b. Additionally or alternatively, the media clip system 106 adds contextual metadata to the second media clip preview 508b. Indeed, in some implementations, the media clip system 106 begins identifying and displaying the content metadata and / or contextual metadata before user confirmation is received to generate a media clip for the content item.

[0136] In example implementations, the second interactive interface 506 may include additional tools and features. For example, in some implementations, the media clip system 106 includes tools for searching a user account (stored locally or remotely) in a client device and / or a media clip library of other content items. In some implementations, the second interactive interface 506 includes an additional area for user annotations or comments that the media clip system 106 adds as contextual metadata when generating a media clip for the content item 502.

[0137] 5B illustrates a client device 400 having a graphical user interface 402, as described above. As shown, the graphical user interface 402 of FIG. 5B includes an application 516 that displays a web page 518 having text, a portion of the text being within the web page 518 and selected as a content item 510. The application 516 corresponds to a web browser displaying the web page 518, although the application 516 may correspond to other types of applications provided herein.

[0138] In response to detecting an interaction with a content item 510 (e.g., highlighting text, selecting highlighted text, and / or opening a pop-up menu), in one or more implementations, the media clip system 106 displays an interactive interface 512, such as a content item menu, having selectable clipping elements 514. In various implementations, the media clip system 106 provides the selectable clipping elements 514 in a different and / or additional selection menu (e.g., a menu accessed from a toolbar) as part of the application or in response to detecting a shortcut key being triggered.

[0139] Further, upon detecting selection of the selectable clipping element 514, the media clip system 106 can generate a media clip for the content item 510. By way of example, the media clip system 106 generates a media clip for the content item 510 and stores the media clip in a media clip library. In some implementations, the media clip system 106 displays a notification that the content item has been saved as a clip. For example, the media clip system 106 displays a media clip preview 520 indicating that the content item 510 has been generated as a media clip and saved in the media clip library. In various implementations, the media clip preview 520 includes the content item 522 and, in some examples, content metadata and / or contextual metadata. In an exemplary implementation, the media clip preview 520 indicates the media clip collection to which the content item 510 is assigned and provides options to move the media clip to a different media clip collection or to create a new media clip collection.

[0140] As described above, the media clip system 106 can add newly generated media clips to one or more media clip collections. To illustrate, Figures 6A-6B show exemplary media clip collections. In particular, Figures 6A-6B show various interactive interfaces for displaying and utilizing digital media clips, according to one or more embodiments. For context, Figures 6A-6B include the client device 400 and graphical user interface 402 introduced above.

[0141] 6A, the graphical user interface 402 includes a first interactive interface 602a that displays a media clip library 606a. As also shown, the media clip library 606 includes a plurality of media clips, each of which includes a content item along with content metadata elements and / or contextual metadata elements. Indeed, each of the media clips in the media clip library 606 may include additional information from its content metadata, contextual metadata, or both.

[0142] Additionally, the first interactive interface 602a also includes a dynamic search query area 604. In one or more implementations, the media clip system 106 provides the dynamic search query area 604 to facilitate searching for content items within a media clip collection, on a user's client device, or within a user account in a content management system. As provided below in connection with FIG. 7, the media clip system 106 can identify media clips that satisfy a search query in one or more implementations based on comparing contextual metadata of content items on the media clip collection.

[0143] 6B , the graphical user interface 402 of the client device 400 displays a second interactive interface 602b having a digital media clip collection 608. As shown, the digital media clip collection 608 includes media clips 610-610 (i.e., digital media clips) from different sources and / or content item types and can be generated from the media clips in the media clip library 606. In particular, the digital media clip collection 608 includes an image clip 610a, an NFT clip 610b, an audio clip 610c, a text clip 610d, a web page clip 610e, a document clip 610f, a video clip 610g, a 3D model clip 610h, and a social media post clip 610i. Thus, in various implementations, the digital media clip collection 608 includes multiple media clips across various media types (e.g., content item types).

[0144] Indeed, the media clip collections maintained by the media clip system 106 are not limited to a particular content item type, but rather flexibly accommodate a range of content item types (and files, as described below). Furthermore, in addition to providing one or more media clip collections that accommodate the display, organization, and utilization of media clips of different content item types, the media clip system 106 also displays content metadata and / or contextual metadata along with the media clips in the media clip collection. In this manner, the media clip system 106 can assist in grouping, organizing, and / or correlating media clips that would otherwise not be related because they have different (e.g., compatible) content item types.

[0145] As mentioned above, FIG. 7 provides additional details regarding searching for and identifying content items based on contextual metadata. In particular, FIG. 7 illustrates searching for digital media clips based on contextual metadata, according to one or more embodiments. As shown, FIG. 7 includes the aforementioned client device 400 and graphical user interface 402. Additionally, the graphical user interface 402 includes an interactive interface 702 having a dynamic search query area 704, a media clip 706, a media clip preview 708, and a shared media clip element 710.

[0146] As described above, the media clip system 106 enables searching for media clips in one or more media clip collections associated with a client device and / or user account in the content management system. In one or more implementations, the media clip system 106 detects a user-provided term (e.g., a query string) entered into the dynamic search query area 704. Based on the term, the media clip system 106 searches the media clip collection for one or more media clips that match the user-provided term in its content metadata.

[0147] In various implementations, the media clip system 106 can search for media clips for a content item based on contextual metadata associated with the content item in each media clip. For example, the media clip system 106 detects media clips where user-provided terms match the contextual metadata of the media clip (i.e., digital media clip).

[0148] In some implementations, the media clip system 106 compares the context of the user-provided terms with the contextual metadata of the media clip to determine a correlation. To illustrate, the dynamic search query area 704 in the interactive interface 702 shows the user-provided terms "white tee shirt" and "added last week." Thus, the media clip system 106 can search the media clip collection for media clips that correspond to white tee shirts and that were clipped within the past seven days and / or calendar week. In an exemplary implementation, the media clip system 106 searches the content item metadata in the metadata database to determine entries related to media clips that satisfy both user-provided terms.

[0149] 7, the media clip system 106 identifies a list of media clips 706 that correspond to the user-provided terms in the dynamic search query area 704. Additionally, the interactive interface 702 includes a media clip preview 708 that shows the content item and content item metadata for the content item. In some implementations, the media clip preview 708 shows the content metadata and / or contextual metadata for the content item that corresponds to the user-provided terms. Additionally, in various implementations, the media clip preview 708 includes a shared media clip element 710 for sharing the media clip with other user accounts.

[0150] In some implementations, the user-provided terms in the dynamic search query area 704 correspond to web-based signals, application-based signals, or OS-based signals. In some implementations, the user-provided terms are a combination of multiple environmental signals (e.g., contextual signals). For example, the user-provided terms indicate one or more open applications and / or client devices (e.g., notebooks, tablets, phones) that the user was utilizing when the media clip was clipped.

[0151] In one or more implementations, the media clip system 106 searches for media clips based on a sample or query media clip. For example, the media clip system 106 receives a request to find media clips with similar contextual metadata. In this manner, the media clip system 106 can identify additional media clips using distinct combinations of application-based, web-based, and / or OS-based signals for the query media clip.

[0152] Additionally, in various implementations, the media clip system 106 can generate and / or modify media clip collections based on the contextual metadata. For example, the media clip system 106 can create a media clip collection that includes media clips that include one or more specific pieces of contextual metadata, such as a specific application-based signal and a specific OS-based signal. Additionally, the media clip system 106 can recommend new media clips to add to the media clip collection, with new media clips having one or more specific pieces of contextual metadata and / or having contextual metadata that matches those in the media clip collection.

[0153] In additional implementations, the media clip system 106 provides the contextual metadata to a machine learning model and / or a neural network to determine connections between media clips that would otherwise go undetected. For example, the media clip system 106 generates and / or utilizes a machine learning model that groups or clusters content items based on the contextual metadata and suggests media clip collections based on the clustered results.

[0154] Additionally, in various implementations, the media clip system 106 can suggest content items to add to a media clip collection based on the media clip's contextual metadata. For example, when browsing for a content item, the media clip system 106 can identify the content item's contextual metadata and compare the contextual metadata with the media clip's contextual metadata stored in a metadata database (or elsewhere). If the media clip system 106 determines a corresponding value that meets a threshold, the media clip system 106 can notify or prompt the user to add the media clip to their media clip collection and indicate why the media clip system 106 suggested a particular recommendation.

[0155]

[0041] Referring now to Figure 8, this figure illustrates capturing and associating additional content items with contextual metadata for a digital media clip, according to one or more embodiments. As shown, Figure 8 includes the aforementioned client device 400 and graphical user interface 402. Further, graphical user interface 402 includes a web page 802 within an application (e.g., a web browser).

[0156] As described above, in various implementations, the media clip system 106 generates a media clip from the clipped content item. In some implementations, when clipping a content item from a web page (or website), the media clip system 106 can capture a visual representation of the website, causing the media clip system 106 to maintain the website's appearance at the time of capture. In this way, if the website changes or if the website's current appearance is temporary (e.g., a temporary branding promotion), the media clip system 106 can generate a "time capsule" that preserves the appearance and, in some examples, one or more features of the web page for future access and / or use. Indeed, in some implementations, the media clip system 106 stores the web page as a media clip in a media clip collection.

[0157] To further illustrate, the media clip system 106 can detect a request to clip a content item within a web page. In various implementations, the media clip system 106 performs operation 810 of capturing the content item as part of a digital media clip, as described above. For example, the media clip system 106 generates a media clip of the content item that includes the identified and / or generated content metadata and the generated contextual metadata.

[0158] Additionally, in one or more implementations, the media clip system 106 performs an operation 812 of determining that the digital media clip is part of a web page. For example, the media clip system 106 determines the URL of the web page 802 from the content metadata and / or contextual metadata of the content item. For example, in some cases, the media clip system 106 analyzes web-based environmental signals to identify a URL from the web page 802 where the content item resides.

[0159] Further, based on determining that the content item belongs to the web page (e.g., the content item has content metadata indicating a URL), the media clip system 106 can identify and collect additional content items on the web page 802. For example, in various implementations, the media clip system 106 performs operation 814 of capturing the additional content items, style, and layout of the web page. In some implementations, operation 814 includes clipping each content item on the web page 802 and saving each content item as a media clip. In some implementations, operation 814 includes capturing the style (e.g., themes, fonts, style sheets, colors, etc.) of the web page 802, code (e.g., HTML, JavaScript, etc.), and layout of each content item in the web page 802 at the time of capture. For example, the layout indicates the proximity of each content item to one another and / or whether the content items are adjacent to or nested within other content items. In various implementations, the media clip system 106 also detects and captures some or all of the web page features, such as whether the web page includes features from a list of known features.

[0160] In an alternative implementation, rather than generating a media clip of some or all of the content item on the web page 802, the media clip system 106 generates and stores contextual metadata for the content item as seen in the current version of the web page by capturing the elements (e.g., the content item), style, and layout that are displayed when extracting the content item from the web page. Indeed, when the content item is later accessed, the media clip system 106 can provide a visual depiction of the website as part of the contextual metadata of the media clip.

[0161] In one or more implementations, the media clip system 106 performs an operation 816 that generates a time capsule of the web page 802. For example, the media clip system 106 utilizes the layout and style to arrange the captured content items in the media clips as they are found and displayed on the web page 802. Indeed, the media clip system 106 can add contextual metadata to one or more of the media clips for the content items from the web page 802 that preserves the layout and style of the website when the content items were captured (e.g., as currently displayed on the website).

[0162] Additionally, in various implementations, the media clip system 106 performs operation 818 of linking the captured web page with the digital media clip. In some implementations, the media clip system 106 generates a time capsule for the web page as part of the contextual metadata of one or more media clips of a content item from the web page. For example, the contextual metadata is stored in a metadata database. In various implementations, the media clip system 106 stores the time capsule for the web page in a segregated database or file structure. In these implementations, the media clip system 106 can link the time capsule or the saved current version of the web page to one or more media clips of the content item from the web page.

[0163] As described above, in some implementations, the media clip system 106 generates a time capsule in connection with generating a media clip for a content item from a web page. In some implementations, the media clip system 106 provides a direct menu option for saving or archiving the current version of a website. For example, the media clip system 106 facilitates a right-click menu with selectable elements such as "Save this web page" or "Create a time capsule for this web page."

[0164] The media clip system 106 can provide various UI / UX features that offer different options for capturing a web page or website. For example, in some implementations, the media clip system 106 can provide an option to capture a static time capsule of a web page or website, meaning that the static time capsule preserves the content, format, and other attributes of the web page or website as they exist at the time of capture. In other examples, the media clip system 106 can provide an option for a user to select to store a live time capsule, meaning that the live time capsule is automatically updated as the underlying web page or website updates. In other words, the media clip system 106 stores an active reference to the live version of the web page or website and periodically updates the stored version of the web page or website in the media clip system 106 to the current version. Furthermore, when a user accesses the live time capsule, the media clip system 106 can access, store, and provide the latest version of the web page or website.

[0165] In one or more additional embodiments, the media clip system 106 may provide an option to time-capture a web page or website. For example, the media clip system 106 may provide selectable options that allow a user to configure how often the media clip system 106 captures versions of a web page or website (e.g., daily, weekly, monthly, yearly, etc.). In this manner, the media clip system 106 may provide tools and interfaces that show how a web page or website changes over a specified period of time. For example, the media clip system 106 may provide an interface that presents different versions of a web page in succession (e.g., playing like a video) to provide a visual representation of the changes. Additionally, the media clip system 106 may analyze two or more different versions of a web page and mark or highlight changes introduced between later versions.

[0166] 8 describes generating functionality that saves the current version of a web page, in an exemplary implementation, the media clip system 106 can apply similar techniques to save the current version of an application and / or generate an interactive interface. Indeed, the media clip system 106 can utilize environmental signals from multiple digital environments to save the current environment or even OS state of an application.

[0167] In some embodiments, the media clip system 106 can remove elements from a website that a user may not want to save. For example, the media clip system 106 can remove advertisements, pop-up elements, or other elements from the web page 802. For example, the media clip system 106 can provide a user-selectable option during the capture process when the media clip system 106 receives the following indication:

[0168] The foregoing figures, corresponding text, and examples provide several different systems, methods, techniques, components, and / or devices of the digital media clip system 106 according to one or more embodiments. In addition to the above description, one or more embodiments may also be described in terms of flowcharts that include operations for achieving a particular result. For example, FIG. 9 shows a flowchart corresponding to one or more implementations of the media clip system 106 described above. Furthermore, the operations outlined in FIG. 9 are provided by way of example only, and some of the operations may be optional, combined into fewer operations, or expanded into additional operations without detracting from the essence of the disclosed embodiments. Furthermore, the operations described herein may be repeated or performed in parallel with each other or with different instances of the same or similar operations.

[0169] Furthermore, the digital media clip system 106 may perform one or more operations in a sequence of operations in addition to, or instead of, one or more operations described in connection with other figures. While FIG. 9 illustrates operations according to one embodiment, alternative embodiments may omit, add, rearrange, and / or modify any of the operations illustrated. Furthermore, each of the operations in FIG. 9 may be performed as part of a method. Alternatively, a non-transitory computer-readable medium may comprise instructions that, when executed by one or more processors, cause a computing device to perform the operations of FIG. 9. In some embodiments, a system may perform the operations of FIG. 9.

[0170] To illustrate, FIG. 9 includes a series of operations 900 for generating a digital media clip in accordance with one or more embodiments. As shown, the series of operations 900 includes an operation 910 for extracting a content item and content metadata. For example, operation 910 may include extracting the content item and content metadata for the content item based on detected user interaction with the content item. In one or more implementations, operation 910 includes detecting a selection of the content item for inclusion in a digital media clip collection. In some implementations, operation 910 includes extracting the content item and content metadata for the content item based on the file format of the content item. In particular implementations, the content metadata includes the name, size, file, type, attributes, tags, creation date, and permissions of the content item.

[0171] Further, the series of operations 900 includes an operation 920 of generating contextual metadata for the content item based on the contextual signals. For example, operation 920 may include generating contextual metadata for the content item based on contextual signals from one or more digital environments from which the content item was extracted. In some implementations, the contextual metadata includes one or more digital environment signals from one or more digital environments associated with extracting the content item.

[0172] In one or more implementations, operation 920 includes generating contextual metadata for the content item to include a capture source and capture time of the content item. In an example implementation, operation 920 includes generating contextual metadata for the content item based on analyzing contextual signals from one or more digital environments associated with a client device from which the detected user interaction is received. In additional implementations, operation 920 includes analyzing an operating system digital environment to identify operating system signals to add to the contextual metadata. In various implementations, operation 920 includes analyzing a web-based environment to identify adjacent content within a vicinity of the content item to add to the contextual metadata.

[0173] In some implementations, operation 920 includes generating contextual metadata for the content item based on analyzing contextual signals from one or more digital environments that the content item resides in. In various implementations, operation 920 includes generating the contextual metadata based on contextual signals from one or more digital environments, including an operating system environment, an application environment, or a web-based environment.

[0174] The series of operations 900 also includes an operation 930 of generating a digital media clip to include the content item, the content metadata, and the contextual metadata. For example, operation 930 may include generating a digital media clip including the content item, the content metadata, and the contextual metadata. In one or more implementations, operation 930 includes generating content item metadata for the content item in a content item attribute database by combining the content item and the contextual metadata in the content item metadata.

[0175] Further, the series of operations 900 includes an operation 940 for identifying a content item based on contextual metadata from the digital media clip. For example, operation 940 can include identifying the content item based on the contextual metadata from the digital media clip. In various implementations, operation 940 includes identifying the content item based on matching the contextual metadata from the digital media clip with user-provided terms. In one or more implementations, operation 940 includes identifying the content item by searching a digital media clip collection containing multiple digital media clips in a user account in a content management system. Further, in an example implementation, the digital media clip collection includes text clips, image clips, video clips, live video clips, audio clips, audiovisual clips, website clips, document clips, or application clips.

[0176] Further, the series of operations 900 includes an operation 950 of displaying the content item along with the content item metadata. For example, operation 9500 may include displaying the content item along with the content item metadata within an interactive graphical user interface. In one or more implementations, operation 950 includes displaying the content item along with the content item metadata accessed from the content item attribute database within the interactive graphical user interface. In various implementations, the interactive graphical user interface includes a selectable option for retrieving the content item from a web-based environment or an operating system environment.

[0177] The series of operations 900 can include one or more additional operations not shown. For example, in various implementations, the series of operations 900 includes an operation of identifying additional digital media clips in a digital media clip collection in a user account in a content management system based on searching a content item attribute database. In various implementations, the series of operations 900 includes an operation of determining that the content metadata includes a URL of a website; an operation of capturing additional content items, layout, and style currently displayed on the website; and an operation of generating a version of the website that displays the content item with the additional content items, such that the version of the website maintains the layout and style of the website when the content items were captured. In some implementations, based on determining that the content metadata includes a URL, the operation includes generating contextual metadata for the content item by saving a version of the webpage that captures the elements, style, and layout that are displayed when extracting the content item from the webpage.

[0178] In some implementations, the series of operations 900 includes generating a new digital media clip collection including the digital media clip and one or more additional digital media clips based on one or more additional digital media clips that share one or more contextual metadata attributes with the contextual metadata of the content item. CONTENT ITEM RECOMMENDATION FOR MEDIA CLIP COLLECTIONS BASED ON CONTEXTUAL METADATA - Patent application

[0179] As described above, in various implementations, the media clip system 106 provides various tools and approaches for suggesting new content items for addition as media clips to a media clip library or media clip collection. For example, the media clip system 106 can identify new content items to suggest as a user performs normal computer activities. For example, the media clip system 106 recommends content items from media clip collections of websites, applications, and / or other user accounts. In some examples, the media clip system 106 provides customized content item suggestions when assisting a user in searching for new content items to add as media clips.

[0180] To further illustrate, Figure 10 shows a schematic diagram of recommending content items for addition to a media clip collection based on contextual metadata, according to one or more embodiments. In particular, Figure 10 includes a series of operations 1000 performed by the media clip system 106 to identify and suggest content items to be made into media clips and added to a media clip collection.

[0181] As shown in FIG. 10 , the series of operations 1000 includes an operation 1002 of providing a media clip library for a user account. As described above, the media clip system 106 can create, provide, store, maintain, modify, and / or remove a media clip library that includes a set of media clips (i.e., digital media clips). Additionally, the media clip system 106 can maintain multiple media clip collections for a user account. For example, the media clip system 106 can organize the media clip collections by theme, category, topic, and / or as defined by the user. In various implementations, the media clip system 106 creates one or more media clip collections based on determining similarities in their contextual metadata and / or content metadata. Further details regarding the generation of media clip libraries and media clip collections are provided above in connection with FIGS. 6A-6B .

[0182] As shown, the series of operations 1000 includes an operation 1004 of detecting a content item that is not included in a media clip library. For example, in various implementations, the media clip system 106 detects a content item within an application (e.g., a web browser, a document, a photo, a news article, etc.) and determines that the content item is not included in a media clip library associated with a user account. As another example, the media clip system 106 identifies content items from media clip collections associated with other user accounts that are not included in a media clip library associated with the user account. Further details regarding identifying potential content items to add to a user account's media clip library are provided below in connection with FIGS. 11 and 12.

[0183] The media clip system 106 can determine whether to recommend a detected content item that is not included in a media clip library associated with a user account. To illustrate, as shown in FIG. 10 , a series of operations 1000 includes an operation 1006 of determining a correlation between the content item and the media clip library based on contextual metadata. For example, the media clip system 106 compares the contextual metadata of one or more media clips in the media clip library of the user account with the identified content item to determine whether a sufficient correlation exists. In some implementations, the media clip system 106 determines a correlation score between each of the identified content items in the media clip library and the media clip (based on their contextual metadata). Further details regarding determining a correlation between a content item associated with a user account and the media clip system are provided below in connection with FIG. 11 .

[0184] 10 , the series of operations 1000 also includes an operation of recommending, by the media clip system 106, generating a digital media clip for the content item and an operation 1008 of adding the digital media clip to a media clip library of a user account. For example, based on the media clip system 106 determining a correlation between the detected content item and a media clip in the media clip library of the user account (or media clip collection), the media clip system 106 can recommend adding the content item as a media clip (and media clip collection) to the media clip library. Upon detecting an interaction confirming the recommendation, the media clip system 106 can generate a digital media clip from the content item and add the newly generated media clip to the media clip library of the user account (and media clip collection), as described above. Further details regarding recommending content items for addition to a media clip library are provided below in connection with FIGS. 11-14.

[0185] As discussed above, Figure 11 provides additional details regarding the media clip system 106 identifying potential content items, determining correlations between the content items and a user account, and providing recommendations for adding the correlated content items as media clips to the user account's media clip library. In particular, Figure 11 illustrates suggesting adding content items to the user account's media clip library based on contextual metadata, according to one or more embodiments. As shown, Figure 11 includes a graphical user interface of a web browser application 1102 that includes various websites (e.g., shown as "Website 1," "Website 2," and "Website 3"), with content from Website 3 (i.e., Shirts.com) currently being shown.

[0186] As shown, the web browser application 1102 includes various content items, including a t-shirt content item 1104 for purchase. The web browser application 1102 also includes a recommendation notification 1106 recommending that the media clip system 106 add the content item 1104 to a media clip library associated with the user account. The web browser application 1102 may include additional content items that are not shown for simplicity.

[0187] 11 also shows a series of operations 1110 that provide additional details regarding how the media clip system 106 determines to recommend a content item 1104. In various implementations, the series of operations 1110 corresponds to the media clip system 106 utilizing a user's account's current media clips as context for suggesting a new media clip to add. While the series of operations 1110 is described in the context of a content item on a website, the same principles and methods apply to content items in other digital environments, such as within a native application.

[0188] In some implementations, the media clip system 106 utilizes a web browser plug-in or extension to detect and recommend content on web pages within a web browser application. In one or more implementations, the media clip system 106 utilizes an application natively installed on the client device (e.g., the content management system application 104) to detect content within the application and / or recommend content at the OS level of the client device. Furthermore, in some implementations, the web browser plug-in and the content management system application communicate with each other to facilitate the management and recommendation of media clips.

[0189] As shown in FIG. 11 , the sequence of operations 1110 includes an operation 1120 of detecting content items not included in the media clip library. As a user interacts with a content provider or content source (e.g., via a web browser or application), the media clip system 106 detects when new content items are discovered. In one or more implementations, the media clip system 106 passively detects content items without explicit direction from the user. In alternative implementations, the media clip system 106 receives an indication (e.g., possibly permission) from the user to monitor user activity and provide media clip suggestions. For example, permission is given to the media clip system 106 to monitor activity from designated sources (e.g., approved websites) and / or within approved applications at designated times.

[0190] Upon detecting the presence of a content item (e.g., an image, text, video, etc.), the media clip system 106 can determine whether the content item is included in a media clip library of a user account. For example, based on following user activity and detecting the content item 1104 in a web browser application 1102, the media clip system 106 determines whether the content item 1104 is included in a media clip library of a user account. If so, the media clip system 106 can provide a notification to the user account that the content item is recognized as belonging to its media clip library. Additionally, the media clip system 106 can update the media clip for the content item by collecting additional content metadata and contextual metadata for the content item.

[0191] Alternatively, if the media clip system 106 determines that the content item 1104 is not included in the user account's media clip library, the media clip system 106 may proceed to operation 1122 in the series of operations 1110. As shown, operation 1122 includes determining a correlation score between the content item and the media clip library based on the contextual metadata. For example, in one or more implementations, the media clip system 106 determines a correlation score for one of the content items by comparing the content item to one or more media clips in the media clip library (or media clip collection). In some examples, the media clip system 106 compares the content item to the media clip's contextual metadata.

[0192] To illustrate, in one or more implementations, the media clip system 106 generates a correlation score by comparing the content item with the content item, content metadata, and / or contextual metadata of media clips in a user account's media clip library. For example, the media clip system 106 can compare the content metadata of the content item with the content item metadata (e.g., content metadata and contextual metadata) and media clips in the media clip library to determine a correlation score based on the number or amount of similarities.

[0193] In some implementations, the media clip system 106 detects and utilizes contextual metadata of a content item to compare the content item to a media clip. For example, the media clip system 106 determines whether the contextual metadata overlaps or shares similar attributes. For example, the media clip system 106 determines a correlation score for a content item based in part on whether similar digital environments and / or contextual signals are detected with the content item as stored for the media clip in a media clip library (or media clip collection).

[0194] In one or more implementations, the media clip system 106 utilizes the content item's contextual metadata to determine a correlation score by comparing the content item's contextual metadata to a set of metrics. For example, the content item's contextual metadata may indicate the frequency of website access, the amount of time active with a content item within an app, or the number of user interactions with a particular content item. In some cases, the metrics are specific to a user with a content item and their past activity patterns. Further, based on the contextual metadata satisfying the metrics in the set of metrics, the media clip system 106 generates a correlation score for the content item.

[0195] As shown, the series of operations 1110 includes an operation 1124 that identifies content items having correlation scores that satisfy a digital media clip correlation threshold (or simply a “correlation threshold”). For example, in various implementations, the media clip system 106 determines which of the content items have correlation scores that are above a minimum correlation score (i.e., the correlation threshold). In some implementations, if multiple content items satisfy the correlation threshold, the media clip system 106 selects the content item with the highest score. In alternative implementations, the media clip system 106 selects each content item that satisfies the correlation threshold. In various implementations, the correlation threshold varies based on the content item type and / or the number of corresponding content items in the media clip collection.

[0196] In various implementations, the media clip system 106 utilizes a content item classification machine learning model to determine the correlation score and / or determine whether a content item is correlated to a user account's media clip library. For example, the media clip system 106 generates content item embeddings for the identified content items (e.g., based on the content item, content metadata, and / or contextual metadata) as well as embeddings for media clips in the media clip library. In additional implementations, the media clip system 106 compares the embeddings (e.g., measuring distances in the embedding space) to determine whether they meet a correlation threshold (e.g., within a predetermined distance). In alternative implementations, the media clip system 106 utilizes other versions of the machine learning model to determine the correlation score between the identified content items and media clips in the user account's media clip library.

[0197] As shown, the series of operations 1110 includes an operation 1126 that suggests adding the content item as a digital media clip to a media clip library. For example, upon determining a correlation between the content item 1104 and one or more media clips in the user account's media clip library, the media clip system 106 provides a recommendation notification 1106, as shown.

[0198] In some implementations, the media clip system 106 provides suggestions in a notification message (e.g., text, email, instant, or push message). In some implementations, the media clip system 106 pops up an interactive interface, such as the recommendation notification 1106 shown in Figure 11. In one or more implementations, the media clip system 106 provides, among other features, an application-based interactive interface that floats over other interfaces and provides suggestions.

[0199] As shown, the series of operations 1110 includes an operation 1128 that generates a new digital media clip from the content item including the content metadata and the contextual metadata. To illustrate, upon presenting the recommendation notification 1106 to the user's client device, the media clip system 106 detects (e.g., directly or via the OS) user confirmation of the recommendation (e.g., selecting a "Yes" element). In response, the media clip system 106 can generate a media clip from the content item, as described above. In particular, the media clip system 106 can generate a media clip that includes the content item, content metadata identified from the content item, and contextual metadata generated from contextual signals from the digital environment in which the content item resides (e.g., the web browser application 1102 and the client device's OS).

[0200] As also shown, the series of operations 1110 includes an operation 1130 that adds the new digital media clip to a media clip library. For example, upon creating a media clip, the media clip system 106 adds the media clip to a media clip library associated with the user account. In various implementations, the media clip system 106 also allows the user to indicate a media clip collection in which to place the new media clip (e.g., by selecting the media clip collection via a drop-down menu).

[0201] In various implementations, the media clip system 106 can recommend adding a content item to a particular media clip collection of a user account. For example, if a user account has multiple media clip collections, the media clip system 106 can determine a correlation score for one or more of the media clip collections. The media clip system 106 can then suggest adding the content item to the media clip collection with the most favorable correlation score.

[0202] In many implementations, the media clip system 106 suggests adding media clips to a user account but does not automatically add the media clips without approval or confirmation by the user account. However, in one or more implementations, the content management system 110 automatically generates media clips from correlated content items and adds them to a media clip collection that is separate from the media clips in the user account's media clip library. In this way, the user can decide whether to move the automatically generated media clip to their media clip library or discard it.

[0203] In some implementations, the media clip system 106 removes automatically generated media clips after an event occurs, for example, after a week or a month, at the end of the week, after a set number of views without being added to a separate media clip collection, or when a set number of media clips have been added to a separate media clip collection, a user-specified event, or a separate event.

[0204] As mentioned above, Figure 12 illustrates identifying potential content items to add to a media clip library (or media clip collection) of a user account. To illustrate, Figure 12 shows a media clip system 106 providing suggestions for adding content items from a media clip library of a second user account to a media clip library of a first user account, according to one or more embodiments.

[0205] As shown, FIG. 12 includes a first user account 1202a (i.e., User A) and a second user account 1202b (i.e., User B). Additionally, FIG. 12 includes a first media clip library 1204a associated with User A and a second media clip library 1204b associated with User B, each media clip collection including a variety of media clips (i.e., digital media clips) that may vary across a wide range of content item types, as provided above. One or more embodiments described with respect to FIG. 12 may rely on a user affirmatively choosing to share data associated with their content, such that the media clip system 106 can generate the functionality described with respect to FIG. 12. If the user does not affirmatively opt in, the content and data associated with the content in the user account remain private. In an additional embodiment, if the media clip system 106 detects that the user has made a content item public, the media clip system 106 may provide the suggestion functionality described with respect to FIG. 12 with reference only to the published content, since by publishing the content, the user has effectively indicated that privacy of that particular content item is not an issue. However, if a user unpublishes a content item, the media clip system 106 no longer uses the content item for the functions described below.

[0206] As shown, the first media clip library 1204a and the second media clip library 1204b each have a media clip for the same content item (i.e., "Classic Tea"), shown as the first media clip 1206. In one or more implementations, the first media clip 1206 includes the same content item in both media clip libraries. Additionally, the first media clip 1206 may also include similar content metadata (based on the content item itself). However, in some implementations, the first media clip 1206 belonging to the first user account 1202a has different contextual metadata than the first media clip 1206 belonging to the second user account 1202b if the digital environment and contextual signals were different when the corresponding content item was captured. In an alternative implementation, if the first user account 1202a shares the first media clip 1206 with the second user account 1202b, the media clips including their contextual metadata may be the same.

[0207] As described above, the media clip system 106 can suggest adding media clips to a user account. In some implementations, the media clip system 106 recommends adding media clips found in the media clip libraries of other user accounts. To illustrate, FIG. 12 includes a first user account 1202a and a second user account 1202b, along with a first media clip library 1204a and a second media clip library 1204b, as introduced above. As shown, the first media clip library 1204a has three media clips, including a first media clip 1206 and a second media clip 1208. The second media clip library 1204b also has three media clips, including the first media clip 1206.

[0208] In various implementations, the media clip system 106 recommends adding media clips from a media clip library (or media clip collection) of one user account to another user account. For example, the media clip system 106 analyzes media clips in a first media clip library to determine whether one or more of the media clips correlate with a second media clip library of another user. For example, in various implementations, the media clip system 106 compares content items and / or their content metadata and contextual metadata from the first user account 1202a with media clips in the second user account 1202b to determine a correlation score, as described above. Then, based on determining media clips and / or content items from the first user account 1202a that have a correlation score that meets a correlation threshold with the media clips in the second user account 1202b, the media clip system 106 determines to recommend the media clips to the second media clip library 1204b.

[0209] More specifically, in some implementations, upon determining that both the first user account 1202a and the second user account 1202b share the first media clip 1206, the media clip system 106 may determine whether the first user account 1202a has additional media clips that correlate to the media clip of the second user account 1202b. For example, the media clip system 106 may identify contextual metadata for the first media clip 1206 and compare it with the contextual metadata of media clips in the first media clip library 1204a. For example, based on the contextual metadata of the first media clip 1206 belonging to user B, the media clip system 106 may identify a capture time and / or capture a location of the first media clip 1206. The media clip system 106 may then compare the capture time and / or capture with the contextual metadata of the media clips in the first media clip library 1204a. In one example, the media clip system 106 may determine that a second media clip 1208 in a first media clip library 1204a was captured at approximately the same time or from the same source as a first media clip 1206 in a second media clip library 1204b. Thus, the media clip system 106 recommends adding the second media clip 1208 from the first media clip library 1204a to the second media clip library 1204b, as indicated by arrow 1210.

[0210] Before comparing the media clip libraries (or media clip collections) of different user accounts, in some implementations, the media clip system 106 determines whether a sufficient nexus or connection exists between the first user account 1202a and the second user account 1202b. For example, the media clip system 106 determines whether the first user account 1202a is linked to the second user account 1202b through a social, professional, or casual association. In some examples, the media clip system 106 determines whether the first user account 1202a and the second user account 1202b belong to the same organization, team, group, and / or whether they are collaborating on a project. In one or more examples, the media clip system 106 determines whether the first user account 1202a is actively communicating with the second user account 1202b (e.g., via text, email, instant message, or application).

[0211] In some implementations, the media clip system 106 determines connections when two or more user accounts share media clips and / or media clip collections. For example, in various implementations, the media clip system 106 enables multiple user accounts to collaborate on a shared media clip collection. In these implementations, the media clip system 106 can determine connections between the user accounts and determine whether additional media clips not in the shared media clip collection should be shared between the user accounts and / or added to the shared media clip collection.

[0212] As another example, for two connected user accounts, the media clip system 106 may detect that the two user accounts are adding media clips to their respective individual media clip collections at approximately the same time. Here, the media clip system 106 may compare the contextual metadata with the media clip system's content item and content metadata to see if sufficient correlation exists. If so, the media clip system 106 may recommend adding one of the media clips added to one user account's media clip collection to the second user account's media clip collection (and thus to each user account's media clip library).

[0213] Indeed, in various implementations, the media clip system 106 can passively identify (e.g., run in the background) content items to suggest to user accounts. For example, when a first user account interacts with a second user account, the media clip system 106 analyzes the second user account's media clip collection or media clip library to determine whether to recommend one or more media clips to the first user. As described above, the media clip system 106 can compare contextual metadata and / or other portions of the media clips to determine the correspondence of the media clips and whether to recommend the content item to the first user account.

[0214] 12 illustrates comparing two user accounts, the media clip system 106 may compare other types of graphs and data structures. For example, the media clip system 106 may compare event graphs, people graphs, content graphs, work graphs, project graphs, social graphs, etc. to determine whether content items and / or media clips in the graphs correspond to media clips in the user account's media clip collection.

[0215] As described above, the media clip system 106 can offer to add customized and personalized media clips to a user account based on the media clip's contextual metadata. To illustrate, Figure 13 shows generating personalized search results for content items based on implicit user preferences, according to one or more embodiments. As shown, Figure 13 includes a search query 1302, initial search results 1304, implicit user preferences 1306, and personalized search results 1308.

[0216] To illustrate, the media clip system 106 can receive a search query from a user associated with a user account on the content management system. For example, a user desiring to discover new content provides search terms to the media clip system 106. In some examples, the search query 1302 is text (i.e., a string of characteristics). In other examples, the search query 1302 is a selected content item and / or media clip, such as a media clip from a media clip collection.

[0217] In various implementations, the media clip system 106 provides an interactive interface for a user to enter a search query 1302. For example, FIG. 7 provides an example of an interactive interface for entering a search query and / or searching for content. FIG. 14, described below, also provides an example of an interactive interface. In some implementations, the media clip system 106 operates in the background of a client device. For example, the media clip system 106 adds a filtering layer to the initial search results of other applications and / or OS systems so that the search results provided to the user are refined based on contextual metadata of media clips belonging to the user account, as described below. In some embodiments, the media clip system 106 provides an option to search all published collections across all published content from various users, e.g., across a team, an organization, or a content management system. Another example of a search option provided by the media clip system 106 is to search only a user's specific collection corpus.

[0218] 13, the media clip system 106 receives a search query 1302 for "movie posters." Accordingly, the media clip system 106 searches for content items related to movie posters. In one or more implementations, the media clip system 106 searches sources outside the user account (e.g., outside the user account's media clips). For example, the media clip system 106 searches repositories of content items / media clips, online sources, and / or other user accounts' media clip collections (e.g., shared or public media clip collections). In some implementations, the media clip system 106 searches through media clips and / or content items stored in association with the user account (e.g., content items stored on the user's client device or in cloud storage).

[0219] As shown, content management system 110 generates initial search results 1304 from search query 1302. As mentioned above, in some implementations, another application or OS system generates initial search results 1304. In many implementations, initial search results 1304 are not provided for display to the user at this time. In various implementations, initial search results 1304 are provided to the user for display.

[0220] 13 further illustrates the content management system 110 identifying implicit user preferences 1306. For example, the media clip system 106 accesses contextual metadata of media clips in a media clip collection and / or retrieves content item metadata from a metadata database (described above) to identify the implicit user preferences 1306. In various implementations, the implicit user preferences 1306 include preferences that are not explicitly expressed by the user but are inferred from analyzing the media clip and / or content item metadata for the user account.

[0221] To illustrate, the media clip system 106 may determine that a user account prefers realistic stock images over cartoon or abstract images. For example, the media clip system 106 may analyze the user account's media clip system to determine this implicit preference for the user account. Other implicit preferences may reveal that the user account prefers a first style of writing (or images) when adding media clips in the morning and a second style of writing (or images) when adding media clips in the evening.

[0222] As another example, in one or more implementations, the media clip system 106 identifies a context signal from the current digital environment on the client device. The media clip system 106 then identifies a subset of media clips in the user account's media clip library that have the same or similar contextual metadata (e.g., date and time, open applications, similar open web browser tabs, etc.). The media clip system 106 then determines implicit user preferences 1306 from the subset of media clips.

[0223] 13 includes personalized search results 1308 that show content items that match the search query 1302 and are specifically tailored to the user account. For example, upon identifying the implicit user preferences 1306, in various implementations, the media clip system 106 applies, filters, and / or compares the implicit user preferences 1306 to the initial search results 1304. In some implementations, the media clip system 106 determines a correlation score (e.g., using machine learning as provided above) based on comparing the initial search results 1304 to the implicit user preferences 1306. Furthermore, the media clip system 106 can select a subset of the initial search results 1304 that are consistent with the implicit user preferences 1306 to generate the personalized search results 1308.

[0224] In various implementations, the media clip system 106 provides the personalized search results 1308 in an interactive interface. In some implementations, the media clip system 106 provides the personalized search results 1308 in a content management system application. Additionally, in one or more implementations, the media clip system 106 provides an option to easily add content items to the personalized search results 1308 to be added to a user account's media clip library.

[0225] Turning to the next figure, Figure 14 provides more details regarding tuning the media clip system 106 and utilizing a media clip classification machine learning model to recommend content items, according to one or more embodiments. In particular, Figure 14 includes a series of operations 1400 performed by the media clip system 106 to provide search results.

[0226] As shown, the series of operations 1400 includes an operation 1410 of receiving a search query and similarity parameters. For example, the media clip system 106 provides an interactive interface 1412 that enables the media clip system 106 to receive user input from a user account regarding searching for content items and / or media clips.

[0227] As shown, the interactive interface 1412 includes a search term section 1414, a media clip selection section 1416, and a similarity scale section 1418. In one or more implementations, the search term section 1414 facilitates user-provided search terms (i.e., query terms). In some implementations, the media clip selection section 1416 facilitates selection of a target media clip or content item, which may also serve as a search query for the media clip system 106. In some implementations, the media clip system 106 includes and / or utilizes both the search term section 1414 and the media clip selection section 1416 in the interactive interface 1412. In other implementations, the client device 102 includes and / or utilizes only one of the search query inputs.

[0228] The interactive interface 1412 also has a similarity scale section 1418. As shown, the similarity scale section 1418 includes a similarity scale indicating how similar or dissimilar the results must be to produce a search result. In effect, the similarity scale section 1418 can represent a range of relevance, where the selected level can range from highly correlated to uncorrelated. In one or more embodiments, the similarity scale 1418 can be a toggle with a larger binary value, e.g., 1 = similar and 0 = dissimilar. Alternatively, as shown in FIG. 14, the similarity scale 1418 can include a slider bar that the user can interact with to set the level of similarity between very similar and very dissimilar. In particular, when the media clip system 106 detects an input that moves the similarity metric (e.g., the relevance level) toward “very similar,” the media clip system 106 attempts to identify content items and / or media clips from the user's media clip collection that are increasingly related to the media clip while also satisfying the search query. Similarly, as the similarity metric is moved toward "very dissimilar," the media clip system 106 attempts to identify content items and / or media clips that are increasingly different from the media clips in the user account's one or more media clip collections while still satisfying or matching the search query.

[0229] Based on the indicated relevance level, the media clip system 106 can modify how it searches for content items. To illustrate, the series of operations 1400 includes an operation 1420 that tunes a media clip classification machine learning model based on a similarity parameter (e.g., an indicated relevance level). For example, in various implementations, the media clip system 106 tunes a media clip classification machine learning model (or simply a “classification model”) to identify content items according to the indicated relevance level and / or content item metadata of media clips in a user account. In some implementations, the media clip system 106 pre-trains different parameter sets of the classification model for different relevance levels (e.g., similarity metric values).

[0230] Further, utilizing the tuned classification model, the media clip system 106 can apply a search query to identify appropriate content item results. To illustrate, the series of operations 1400 includes an operation 1430 that utilizes the tuned media clip classification machine learning model to identify an existing system. Indeed, in various implementations, the media clip system 106 provides search terms and / or target media clips (or target content items) from the interactive interface 1412 to the tuned classification model to identify one or more content items based on the search query, while also identifying content items that may or may not be related to the user account's current media clip (e.g., based on the content item metadata of the media clip).

[0231] As shown, the series of operations 1400 includes an operation 1440 of providing search results. In particular, for content item results that are similar to the media clips in the user account, the media clip system 106 can provide similar results 1442. For content item results that are different from the media clips in the user account, the media clip system 106 can provide different results 1444. Additionally, the media clip system 106 can provide a range of content item results in between based on the relevance level selected by the user account.

[0232] Indeed, in various implementations, the media clip system 106 facilitates users' searches for new content items that can range widely and flexibly from content items that match content items in a user's account's current media clip collection to content items that are significantly different from content items in a user's account's current media clip collection. In this manner, the media clip system 106 provides additional functionality not offered by existing systems when searching for and adding content items to a user's account's media clip collection.

[0233] The media clip system 106 can also provide additional functionality with respect to media clips and contextual metadata that can be of great help to users. To illustrate, in various implementations, the media clip system 106 provides additional suggestions in addition to adding content items to a media clip collection. For example, in various implementations, the media clip system 106 utilizes contextual signals to determine that a user is filling out a form. Furthermore, based on matching one or more contextual signals with contextual metadata of a media clip, the media clip system 106 can suggest entering information from the media clip into the form.

[0234] In particular, in various implementations, the media clip system 106 detects that a user has filled out a form that requires personal information, such as a driver's license number. Based on analyzing context signals (e.g., application-based signals) associated with the form, the media clip system 106 can identify a corresponding media clip, such as an image or scanned image of the user's driver's license. Additionally, in some examples, the media clip system 106 extracts the driver's license number from the media clip (if not previously extracted and stored as content item metadata) and suggests entering the driver's license number into the form.

[0235] The foregoing figures, corresponding text, and examples provide several different systems, methods, techniques, components, and / or devices of the digital media clip system 106 according to one or more embodiments. In addition to the above description, one or more embodiments may also be described in terms of flowcharts that include operations for achieving a particular result. For example, FIG. 15 shows a flowchart corresponding to one or more implementations of the media clip system 106 described above. Furthermore, the operations outlined in FIG. 15 are provided only as examples, and some of the operations may be optional, combined into fewer operations, or expanded into additional operations without detracting from the essence of the disclosed embodiments. Furthermore, operations described herein may be repeated or performed in parallel with each other or with different instances of the same or similar operations.

[0236] Furthermore, the digital media clip system 106 may perform one or more operations in a sequence of operations in addition to, or instead of, one or more operations described in connection with other figures. While FIG. 15 illustrates operations according to one embodiment, alternative embodiments may omit, add, rearrange, and / or modify any of the illustrated operations. Furthermore, each of the operations in FIG. 15 may be performed as part of a method. Alternatively, a non-transitory computer-readable medium may include instructions that, when executed by one or more processors, cause a computing device to perform the operations of FIG. 15. In some embodiments, a system may perform the operations of FIG. 15.

[0237] 15 includes a series of operations 1500 for recommending content items for addition to a media clip library based on contextual metadata, in accordance with one or more embodiments. As shown, the series of operations 1500 includes an operation 1510 for maintaining a media clip library for a user account. For example, operation 1510 may include maintaining, for a user account in a content management system, a media clip library including content items, content metadata, and digital media clips, each having contextual metadata.

[0238] Further, the series of operations 1500 includes an operation 1520 for detecting a content item. For example, operation 1520 may include detecting a content item that is not associated with a user account. In one or more implementations, operation 1520 may include identifying a plurality of content items that are not associated with a user account that includes the content item. In some implementations, operation 1520 may include identifying a plurality of content items from an additional media clip library (or media clip collection) of an additional user of the content management system.

[0239] The series of operations 1500 also includes an operation 1530 for determining a correlation score between the content item and the media clip. For example, operation 1520 may include determining a correlation score between the content item and one or more digital media clips in a media clip library. In one or more implementations, operation 1530 may include selecting a content item from the plurality of content items based on the correlation score. In some implementations, operation 1530 may include determining a correlation between contextual metadata of the digital media clip and contextual metadata from a second media clip library of a second user account in the content management system. In various implementations, operation 1530 may include selecting the content item based on the correlation score of the content item that meets a digital media clip suggestion threshold. In an example implementation, operation 1530 may include determining a correlation between the content item and one or more digital media clips in the media clip library by comparing contextual metadata associated with the content item with contextual metadata of the digital media clips in the media clip library.

[0240] Further, the series of operations 1500 includes an operation 1540 of providing a recommendation for generating a new media clip for the content item in the media clip library. For example, operation 1540 may include providing a recommendation for generating a digital media clip for the content item in the media clip library based on the correlation score. In various implementations, operation 1540 may include providing an additional recommendation to the second user account for adding a digital media clip to a second media clip library associated with the second user account based on the correlation.

[0241] Further, the series of operations 1500 includes an operation 1550 that generates a digital media clip for the content item. For example, operation 1550 may include generating a digital media clip for the content item based on receiving user confirmation of the recommendation. In various implementations, operation 1540 may include adding the digital media clip to a media clip library for the user account. In one or more implementations, operation 1540 may include adding the new digital media clip to the media clip library by generating the new digital media clip from the content item based on the correlation (e.g., the correlation score). In some implementations, operation 1540 may include capturing the content item, extracting content metadata from the content item, and generating contextual metadata from contextual signals corresponding to one or more digital environments in which the content item currently resides.

[0242] The series of operations 1500 may also include operations not included in Figure 15. To illustrate, in various implementations, the series of operations 1500 includes the additional operation of determining, for a digital media clip in the media clip library of the user account, that a copy of the digital media clip is located in a second media clip library of a second user account in the content management system, the additional operation of determining an additional correlation between the additional digital media clip in the second media clip library of the second user account and the digital media clip in the media clip library of the user account, and the additional operation of providing a content item from the additional digital media clip in the second media clip library of the second user account as a content item to the user account based on the additional correlation.

[0243] In some implementations, the series of operations 1500 includes an additional operation of identifying a plurality of content items not associated with a user account based on the search query; an additional operation of determining one or more implicit user preferences based on contextual metadata of digital media clips in the media clip library; an additional operation of filtering the plurality of content items based on the one or more implicit user preferences to identify a subset of the content items; and an additional operation of providing the subset of the content items in response to the search query.

[0244] In one or more implementations, the series of operations 1500 includes the additional operations of identifying a target digital media clip, receiving user input indicating a relevance level ranging from high correlation to no correlation, adjusting (tuning) parameters of a media clip classification machine learning model based on the relevance level, and identifying one or more content items utilizing the adjusted media clip classification machine learning model and the target digital media clip. Canvases and Collections

[0245] As described above, content management system 110 facilitates the creation, management, utilization, publication, and presentation of media clip collections. As provided further below, content management system 110 provides tools for generating media clip collections, including adding suggested digital media clips, arranging media clip collections in various layouts, sharing media clip collections, and controlling access to media clip collections via control lenses.

[0246] To illustrate, Figure 16 shows a schematic diagram of generating a media clip collection using various control lenses and layout configurations, according to one or more embodiments. In particular, Figure 16 includes a series of operations 1600 performed by the media clip system 106 to generate a media clip collection, utilize the media clip collection, and present the media clip collection to various viewers using various control lenses and layout configurations.

[0247] 16, the sequence of operations 1600 includes an operation 1602 of generating a media clip collection having digital media clips. As described above, in various implementations, the media clip system 106 enables a user account to generate media clips (i.e., digital media clips) from content items and add them to the user account's media clip library. Additionally, in one or more implementations, the media clip system 106 enables a user account to generate a media clip collection from the media clips in the media clip library. Further details regarding generating a media clip collection are provided below in connection with FIGS. 17A-17F.

[0248] As also shown, the series of operations 1600 includes an operation 1604 that identifies various control lenses to apply to the media clip collection. For example, in various implementations, the media clip system 106 provides several control lenses to apply to the media clip collection. In one or more implementations, the control lenses modify the visual appearance of one or more media clips in the media clip collection, such as adding a watermark or color filter. In some implementations, the control lenses modify access rights to the media clip collection, such as preventing viewing, sharing, or downloading of a media clip, as well as enabling purchase of a media clip. In each case, the media clip system 106 utilizes control lenses to modify the appearance of or access to the media clip collection, while the media clip system 106 leaves the original or underlying media clips in the media clip collection unchanged. Further details regarding the utilization of control lenses are provided in connection with FIGS. 20A-20C below.

[0249] As shown, the series of operations 1600 includes an operation 1606 of sharing different versions of a media clip collection utilizing different control lenses. For example, in various implementations, the media clip system 106 provides a media clip collection with a first set of control lenses to a first viewer, while providing the same media clip collection with a second set of control lenses to a second viewer. In this manner, the two viewers experience the media clip collection differently as affected by the different sets of control lenses. Further details regarding sharing a media clip collection utilizing control lenses are provided below in connection with Figures 20A-20C.

[0250] As also shown, the series of operations 1600 includes an operation 1608 that identifies various layouts for arranging digital media clips. For example, in various implementations, the media clip system 106 provides several layout configurations for displaying a media clip collection. Examples of layout configurations include a grid layout, a gallery layout, a timeline layout, and a freeform layout (e.g., a canvas layout), among other layout configurations. Further details regarding utilizing layout configurations are provided in connection with Configurations 19A-19C below.

[0251] As shown, the series of operations 1600 includes an operation 1610 that provides suggestions for digital media clips in the digital canvas. In various implementations, when a media clip is added to a media clip collection, the media clip system 106 can utilize content item metadata to provide insight into the media clip and suggest related media clips. Further details regarding content management systems that provide suggestions for digital media clips are provided below in connection with Figures 17B-17D.

[0252] As noted above, Figures 17A-17F provide additional details regarding the creation of a media clip collection. Specifically, Figures 17A-17F illustrate the creation of a media clip collection according to one or more embodiments. To illustrate, Figure 17A includes an interactive interface 1700a on a client device 400, displayed within the graphical user interface 402 described above.

[0253] As shown, the interactive interface 1700a includes a media clip library 1702a and a media clip 1706a, also described above. In some examples, the interactive interface 1700a is part of an application or is a short-term floating interface triggered by a shortcut key or a detected menu selection. Additionally, the graphical user interface 402 also includes a digital canvas 1710a for creating a media clip collection. For example, each of the media clips 1706a added to the digital canvas 1710a constitutes a media clip collection. In various implementations, the media clip system 106 implements the digital canvas 1710a within an application 1708a, such as a native application (e.g., content management system application 104) on the client device 400 or within a web browser application.

[0254] As also shown, the digital canvas 1710a includes a media clip collection title 1712a for labeling the media clip collection. Additionally, the digital canvas 1710a includes one or more media clips 1716a. For example, the media clip system 106 allows a user account to add one or more of the media clips 1706a from a media clip library 1702a in the interactive interface 1700a to the digital canvas 1710a. In many implementations, the media clip system 106 allows a user account to select and drag one of the media clips 1706a from the interactive interface 1700a to the digital canvas 1710a without leaving the digital canvas 1710a, switching interfaces, or switching tabs (when the digital canvas 1710a is implemented in a web browser).

[0255] The media clip system 106 can enable user accounts to organize media clips 1716a within the digital canvas 1710a as needed. Various configurations for organizing media clips 1716a are provided below with respect to Figures 19A-19C. Additionally, in various implementations, the media clip system 106 can provide tools related to the digital canvas 1710a, such as a digital whiteboard, flow charts, note-taking, and other editing tools.

[0256] In one or more implementations, when adding a media clip to a media clip collection, the media clip system 106 adds a representative media clip, a symbolic media clip, or a placeholder media clip to the media clip collection (i.e., media clip 1716a in digital canvas 1710a is a placeholder media clip for media clip 1706a). Indeed, rather than duplicating the media clip, the media clip system 106 represents a placeholder for the media clip in the media clip collection while maintaining a single copy of the media clip in a media clip library (e.g., stored in the media clip database and / or another database). In this way, a user account can have the same media clip in several media clip collections without requiring additional storage space on the client device 400 for each version or copy of the media clip.

[0257] 17B illustrates another embodiment for creating a media clip collection. As shown, FIG. 17B includes a client device 400 having a graphical user interface 402. As shown, the graphical user interface 402 includes a web browser application 1708b that displays a digital canvas 1710b and a media clip collection title 1712b. Additionally, the web browser application 1708b includes a side panel 1704 that includes a media clip library 1702b having media clips 1706b, files 1718, and suggested media clips 1722. In various implementations, the side panel 1704 is part of a web browser plug-in that appears when a selection thereof is detected.

[0258] As shown, the media clip system 106 facilitates a user account in building a media clip collection by moving one or more of the media clips 1706b to the digital canvas 1710b. For example, the media clip system 106 detects a user account that adds media clips from the media clip library 1702b, shown as metadata 1716b, to the digital canvas 1710b. In this manner, the media clip system 106 facilitates building a media clip collection from the media clips in the user account's media clip library.

[0259] In various implementations, the media clip system 106 provides additional tools for creating a media clip collection. For example, the side panel 1704 includes tools for searching or filtering media clips. As introduced above, the side panel 1704 also includes files 1718 and recommended media clips 1722 (described in the next figure). In various implementations, a media clip collection can include files in addition to media clips. Files can include documents, images, text, video, and other stored data on the client device 400 (or stored remotely) that are not included in the media clip library of the user account or another user account in the content management system. In effect, a media clip collection flexibly enables files and media clips to be viewed together in a dynamic and seamless manner.

[0260] While FIG. 17B illustrates a desktop embodiment of the graphical user interface 402, the media clip system 106 may also provide a mobile graphical user interface. For example, in one or more embodiments, the media clip system 106 may provide a graphical user interface comprising a mobile share sheet for display on a mobile terminal (e.g., a smartphone). The mobile share sheet may include an area within the graphical user interface where a user can drag and drop content items to store and / or share content items via a collection, as described in detail herein. For example, the media clip system 106 may provide an area where a user can clip a content item (e.g., an image) from a web page, drag the clipped content item into the area, and drop the content item within the area. Dropping the content item within the share sheet saves the content item to a collection with the content management system.

[0261] As described above, in various implementations, the media clip system 106 can provide suggestions for media clips. To illustrate, FIG. 17C shows a media clip 1720 in an application 1708c (e.g., a web browser). In one or more implementations, the media clip 1720 is part of a media clip collection, such as a media clip recently added to the media clip collection, as described above. In an alternative implementation, the media clip 1720 is a media clip selected from a media clip library of a user account. In either case, as shown in FIG. 17C, the media clip system 106 updates the application 1708c to focus on the media clip 1720.

[0262] As shown, the application 1708c includes a media clip 1720, a title for the media clip 1720, a media clip description 1726, and displayed media clip metadata 1728. In various implementations, the media clip description 1726 includes a brief description of the media clip 1720 and may include a user-provided description, information obtained from content metadata and / or contextual metadata, and / or information automatically generated based on the media clip. Similarly, the displayed media clip metadata 1728 may indicate various metadata elements from the content item metadata (e.g., content metadata and / or contextual metadata) for the media clip. Additionally, although not shown, the media clip metadata 1728 may include a link to the original source of the media clip 1729. For example, the link may be to a document in a content management system or a web page from which the media clip 1729 was captured. As shown, the displayed media clip metadata 1728 indicates the creator, location, format, and year of the media clip 1720. In various implementations, the metadata elements shown in the displayed media clip metadata 1728 are based on the format (eg, file type) of the media clip being provided.

[0263] In some implementations, the media clip description 1726 displays insights. For example, the media clip system 106 utilizes machine learning models and / or other analyses to discover less common genres, categories, typefaces, or another topic for the media clip (e.g., "The aesthetic of this clip is called 80s grunge" or "This typeface is called Gothic Bold"). Additionally, in one or more implementations, the media clip system 106 provides follow-up actions to the suggestions, such as adding notes to the media clip or initiating a search based on the insights. In an example implementation, the media clip system 106 provides the insights in other locations (e.g., within the digital canvas and / or media clip collection), such as in a pop-up interface that appears when the media clip is added or when a user account interacts with the media clip.

[0264] As shown, the application 1708c also includes sections for similar media clips 1724 and suggested media clip descriptions 1726. In various implementations, the media clip system 106 identifies similar media clips 1724 by searching through the user account's media clip library. Additionally, the media clip system 106 can search the media clip libraries of other user accounts within the content management system, including public shared media clip libraries, as well as user accounts that have granted the user account permission to connect and contribute clips from a community 1725, as shown in FIG.

[0265] In one or more implementations, the media clip system 106 determines similar media clips by comparing content item metadata between the media clip 1720 and other media clips. For example, as described above, the media clip system 106 compares the content metadata and / or contextual metadata of the media clip 1720 with similar metadata of the other media clips to determine a correlation score. In various examples, the media clip system 106 then provides the highest-ranked media clips within the application 1708c as suggested similar media clips, as shown. In various implementations, the media clip system 106 detects the selection of the suggested media clip and enables user accounts to add the suggested media clips to their media clip libraries and / or media clip collections.

[0266] As further shown in FIG. 17C , the graphical user interface may include a notes section and a comments section. For example, when a user adds a media clip 1720 to a collection, the media clip system 106 may provide a notes section that allows the user to save the media clip 1720 and enter notes. Thus, when the media clip system 106 provides access to the media clip 1720 to a client device, the media clip system 106 may provide the notes that were originally saved when the user clipped the media clip 1720. Additionally, as further shown in FIG. 17C , the graphical user interface 402 may include a comments section that displays comments from various users with whom the media clip 1720 has been shared. In this manner, by simply clipping an item, the media clip system 106 creates a collaborative environment that allows users to view, comment on, and use the media clip 1720.

[0267] In addition to providing suggested media clips, in various implementations, the media clip system 106 can suggest a media clip collection from media clips in a media clip library of a user account. To illustrate, FIG. 17D shows the media clip system 106 suggesting multiple suggested media clip collections. As shown, FIG. 17D includes an interactive interface 1700b that includes a media clip library 1702a and a media clip 1706a. The interactive interface 1700b also includes a suggested media clip collection 1730. For example, the content management system 110 updates the interactive interface 1700a from FIG. 17A by adding the suggested media clip collection 1730 to display the interactive interface 1700b, as shown in FIG. 17D.

[0268] In various implementations, the media clip system 106 generates the suggested media clip collection based on determining correspondences between content item metadata of media clips in a user account's media clip library. For example, the media clip system 106 determines groups of media clips that share similar combinations of contextual signals and / or contextual metadata, revealing combinations that would not otherwise be revealed. In some implementations, the media clip system 106 utilizes a machine learning model (e.g., a media clip classification machine learning model) to generate an embedding space of media clips, and the media clip system 106 generates the suggested media clip collection by clustering media clip embeddings within the embedding space.

[0269] As shown, in various implementations, the media clip system 106 provides a suggested media clip collection 1730 to a user account in the interactive interface 1700b. In some implementations, the media clip system 106 otherwise provides one or more suggested media clip collections 1730 to the user account. In response, the user account can choose to keep or ignore the suggested media clip collection. For example, in response to providing the suggested media clip collection, the media clip system 106 detects a user interaction that adds the media clip collection to the user account.

[0270] To illustrate, in various implementations, the media clip system 106 provides a display of multiple media clip collections. For example, FIG. 17E shows a client device 400 having a graphical user interface 402 that illustrates an interactive interface 1700d. Additionally, the interactive interface 1700d illustrates a digital media collection library 1732 that includes multiple media clip collections. In various implementations, the media clip system 106 provides tools for searching through the media collection library 1732, as well as tools for modifying, copying, deleting, sharing, and / or otherwise managing media clip collections associated with a user account. As can be seen in FIG. 17E, the media clip collection provides a display of a wide variety of different file types and media items visualized together in a single graphical user representation of the collection.

[0271] In some implementations, a media clip collection can contain other media clip collections. For example, the media clip system 106 embeds a media clip collection (or a representation of a media clip collection) within another media clip collection. In some implementations, the media clip system 106 allows media clip collections to be grouped with other media clip collections, such as workspaces and personal groups.

[0272] Additionally, the media clip system 106 can provide additional details for individual media clip collections. To illustrate, FIG. 17F shows the media clip system 106 displaying a digital media collection. As shown, FIG. 17F includes a client device 400 having a graphical user interface 402 that includes an application 1708d. As mentioned above, the application 1708d can represent an application native to the client device 400 or a web browser application. In some implementations, the media clip system 106 replaces one or more of the illustrated applications with the aforementioned interactive interface.

[0273] As shown, application 1708d includes a media clip collection title 1712c, a media clip 1716c, a media clip collection description 1734, and metadata 1736. In various implementations, media clip 1716c is included as part of a digital canvas that allows a user to modify the media clips in the media clip collection.

[0274] In one or more implementations, the media clip collection description 1734 provides a brief description of the media clip collection. For example, the media clip system 106 automatically generates one or more lines of text that encapsulate the media clips included in the media clip collection. For example, the media clip system 106 utilizes a machine learning model or other summary generation model to process the content item metadata of the media clips in the media clip collection and generate the media clip collection description 1734.

[0275] Similarly, in various implementations, the media clip system 106 compiles and provides metadata 1736 in association with the media clip collection. In some implementations, the metadata 1736 includes metadata elements similar to the media clip metadata 1728 displayed above in FIG. 17C, but for some or all of the media clips in the media clip collection. For example, the metadata 1736 indicates multiple formats of the media clips and / or files included in the media clip collection. Additionally, as shown in FIG. 17F, the graphical user interface 402 can include a comments section that allows users to comment on the content of the collection to enable user collaboration within the collection.

[0276] As described above, in various implementations, the media clip system 106 can add both media clips and files to a media clip collection. In some examples, the media clip system 106 can also add stacked digital media clips (or simply "stacked media clips"). In various implementations, the media clip system 106 generates a stacked media clip 1802 by binding a set of digital media clips in a defined presentation order. Further details regarding stacked media clips are provided with respect to Figures 18A-18B.

[0277] To illustrate, Figures 18A-18B show stacked media clips according to one or more embodiments. As shown, both Figures 18A-18B include a client device 400 having a graphical user interface 402, as introduced above. Additionally, both Figures 18A-18B include an application 1804, which is shown as a web browser. However, in other implementations, the application 1804 is a native app and / or an interactive interface as described above.

[0278] As shown, Figure 18A illustrates a media clip collection 1810 including a media clip 1806 that includes a stacked media clip 1802. In particular, the stacked media clip 1802 includes a first stacked media clip 1802a and a second stacked media clip 1802b. Although the stacked media clip 1802 in the media clip collection 1810 illustrates two media clips joined or spliced ​​together, the media clip system 106 can generate stacked media clips that include more than two media clips. Indeed, the media clip system 106 can detect a selection of multiple media clips along with a request to generate a stacked media clip from the media clips. In an additional implementation, the media clip system 106 also receives a presentation order of the media clips in the stacked media clip.

[0279] In various implementations, because the first stacked media clip 1802a and the second stacked media clip 1802b are spliced ​​into the stacked media clip 1802, the media clip system 106 displays the stacked media clip 1802 as a single entity within the media clip collection. In various implementations, the media clip system 106 does not show the first stacked media clip within the stacked media clip without showing at least a portion of the second stacked media clip. In this manner, the stacked media clip signals that two or more media clips are designated to move and are presented together as a single entity. Often, by generating and utilizing stacked media clips, the media clip system 106 can provide better context to the viewer or system than either media clip alone would provide. In some implementations, rather than showing at least a portion of each media clip within the stacked media clip, the media clip system 106 indicates, when displayed within the media clip collection, that the media clip collection is a stacked media clip including multiple spliced, combined, or otherwise linked media clips.

[0280] In various implementations, the stacked media clips include a predefined presentation order and / or configuration. To illustrate, Figure 18B shows an application 1804 updating from showing a media clip collection 1810 to showing stacked media clips 1802. In particular, the application 1804 shows the first stacked media clip 1802a and the second stacked media clip 1802b as separate media clips. In some implementations, the application 1804 also includes additional elements such as media clip descriptions, displayed media clip metadata, similar media clips, and suggested media clips, as described above.

[0281] As shown, the media clip system 106 arranges the first stacked media clip 1802a first (on the left) and the second stacked media clip 1802b second (on the right). In various implementations, the media clip system 106 maintains this presentation order when displaying the first stacked media clip 1802a and the second stacked media clip 1802b as separate but combined media clips. In alternative implementations, the media clip system 106 presents the media clips within the stacked media clips in a different order or arrangement.

[0282] In some implementations, the media clip system 106 generates the stacked media clip 1802 by creating a media clip collection of a first stacked media clip 1802a and a second stacked media clip 1802b. For example, the media clip system 106 adds the first stacked media clip 1802a and the second stacked media clip 1802b to the media clip collection. Furthermore, the media clip system 106 applies a layout configuration and / or control lens (described further below) that locks the presentation order of the media clips in the media clip collection. Furthermore, the media clip system 106 adds the stacked media clip / media clip collection to another media clip collection, ensuring that each of the media clips in the stacked media clip is at least partially visible in the other media clip collection.

[0283] As described above, the media clip system 106 can apply various layouts to a media clip collection. For example, the media clip system 106 can generate and / or provide a combination of layout configurations for viewing, editing, displaying, and / or presenting a media clip collection. In particular, Figures 19A-19C illustrate various layout configurations for a media clip collection, according to one or more embodiments.

[0284] As shown, Figure 19A provides an overview of various media clip collection layouts 1900. Specifically, the various media clip collection layouts 1900 range from rigid (left) layout configurations to flexible (right) layout configurations. For example, the various media clip collection layouts 1900 include a basic grid layout 1902, a masonry grid layout 1904, a gallery wall layout 1906, a tabletop layout 1908, a timeline layout 1910, and a canvas layout 1912 (also known as a freeform layout). Although not shown, the various media clip collection layouts 1900 can include other layouts such as a column layout, a hub-and-spoke layout, a flow layout, etc.

[0285] As shown, various media clip collection layouts 1900 provide various layout configuration options for organizing media clips within a media clip collection. Each of the layout configurations is described in further detail below in FIG. 19B. For example, FIG. 19B shows visual examples of layout configurations of a media clip collection with media clips. Although not shown, in some examples, a media clip collection can also include files and stacked media clips in addition to media clips.

[0286] As shown, the media clip system 106 can provide a basic grid layout 1902. In various implementations, the basic grid layout 1902 provides a rigid structure for organizing media clips within a media clip collection. In some implementations, the media clip system 106 provides a masonry grid layout 1904. As shown, the masonry grid layout 1904 provides additional flexibility over the basic grid layout 1902 by allowing various media clips to span multiple grid spaces (or portions of a grid space). For example, in some implementations, the masonry grid layout 1904 allows media clips to span a 1×1.5 (width×height) grid space, as shown. In various implementations, the masonry grid layout 1904 allows media clips to span a 2×2 or 2×3 grid space. In effect, the masonry grid layout 1904 allows media clips to chart the architecture of a basic grid, while allowing additional flexibility to accommodate media clips of different shapes and / or to highlight more important media clips.

[0287] In addition, FIG. 19B illustrates a gallery wall layout 1906. In various implementations, the gallery wall layout 1906 provides a set presentation of media clips in a media clip collection. For example, the gallery wall layout 1906 can define the order in which the media clips are presented, such as forcing scrolling or viewing of the media clips from left to right. As shown, the gallery wall layout 1906 can allow for multiple rows of media clips while still enforcing navigation constraints. In some implementations, the gallery wall layout 1906 allows a user account to have a single row of media clips.

[0288] As shown, the various media clip collection layouts 1900 include a tabletop layout 1908. In various implementations, the tabletop layout 1908 allows media clips, files, and stacked media clips to be spatially arranged while still loosely maintaining a grid architecture. In many examples, the tabletop layout 1908 allows groups or clusters of media clips (media clips can be clustered and overlapping within a media clip collection, unlike stacked media clips) to be presented in an orderly and edited manner.

[0289] 19B further illustrates a timeline layout 1910. In exemplary implementations, the timeline layout 1910 provides a temporally delineated organization of the media clips. For example, the media clip system 106 groups the media clips by the date and time they were added to the media clip collection. In some implementations, the media clip system 106 utilizes the time they were added to the media clip library, the time the media clip collection was created, or another time determined from the content item metadata of the media clips in the media clip collection. In some implementations, the media clip system 106 partitions the timeline layout 1910 by time delineation, such as hours, days, months, years, etc.

[0290] 19B further includes a canvas layout 1912, also referred to as a freeform layout. As shown, the canvas layout 1912 provides a flexible, freeform configuration that allows for customized organization of media clips within a media clip collection. For example, under the canvas layout 1912, a user account can create various groups and clusters of media clips together as needed. Further details regarding the canvas layout 1912 are provided below with respect to FIG. 19C.

[0291] 19C shows a canvas layout 1912 of a media clip collection 1916 with media clips 1918. As shown, the media clip system 106 presents the media clip collection 1916 in an interactive interface 1914, which is further described above. In an alternative implementation, the media clip system 106 presents the media clip collection 1916 in an application and / or a web browser.

[0292] As shown, the media clip collection 1916 is arranged in a canvas layout 1912 on the digital canvas. Thus, the media clips 1918 in the media clip collection 1916 are arranged in a free-form configuration rather than a grid pattern or architecture. Additionally, the media clip collection 1916 includes additional media clip types such as notes (e.g., handwritten notes) and emoji stickers that are clustered with various media clips in the media clip collection 1916. Additionally, the media clip collection 1916 includes stacked media clips 1920.

[0293] In one or more implementations, the canvas layout 1912 (and other layout configurations) can dynamically update the digital canvas to accommodate media clips as they are added to the media clip collection. For example, in some implementations, the media clip system 106 dynamically expands the size of the digital canvas to accommodate additional media clips or as media clips are moved (e.g., expanding the edges of the canvas layout 1912 as media clips are placed nearby). Similarly, as media clips are consolidated and / or removed, the media clip system 106 can efficiently reduce the size of the digital canvas. Furthermore, because media clips in a media clip collection are often placeholders (e.g., the media clips are stored outside of the digital canvas), the media clip system 106 can dynamically expand the digital canvas to large areas and sizes while keeping resource consumption low on the client device.

[0294] Additionally, in various implementations, the media clip system 106 utilizes the canvas layout 1912 as an interactive medium to support user accounts in building their media clip collections. For example, the media clip system 106 can provide suggestions as media clips are added to or interacted with within the canvas layout 1912, as described further below. Indeed, the media clip system 106 can incorporate dynamic intelligence into the canvas layout 1912 to facilitate the ingestion of media clips and files and provide proactive suggestions to guide the creative journey of the user account.

[0295] In addition to providing layout configurations for media clip collections, the media clip system 106 can also provide control lenses to media clip collections that change the appearance of and / or access to the media clip collections without modifying the digital media clips within the media clip collection. To illustrate, Figures 20A-20C show the use of various control lenses in accordance with one or more embodiments.

[0296] In particular, Figure 20A includes a client device 400 and a graphical user interface 402 displaying an application 2004 that includes a media clip 2006 within a media clip collection 2010. As shown, the application 2004 also includes selectable elements that include a share element 2008. In various implementations, upon detecting selection of the share element 2008, the media clip system 106 provides various sharing options, such as publishing the media clip collection (discussed below in Figure 21), sending a link for the media clip collection, or adding a control lens to the media clip collection (e.g., via one or more control lens elements 2012).

[0297] In various implementations, the media clip system 106 generates control lenses for modifying or changing the visual appearance of one or more of the media clips in the media clip collection. For example, the media clip system 106 creates a control lens that adds a watermark or a color filter to the media clips in the media clip collection. In some implementations, the content management system 110 generates control lenses for enforcing access rights to the media clip collection. For example, the media clip system 106 creates a control lens that restricts the viewing, sharing, or downloading of media clips in the media clip collection. In some examples, the media clip system 106 generates control lenses related to facilitating the redemption of media clips in the media clip collection.

[0298] In particular, the control lens element 2012 illustrates various control lens elements, including an appearance control lens element, an access control lens element, and a pop-up shop control lens element. Upon detecting a selection of one of these or another control lens element, the media clip system 106 can apply the selected control lens element to the media clip collection 2010. Each of these control lens options is further described below with respect to FIG. 20B.

[0299] In some implementations, the media clip system 106 applies a control lens to a version (e.g., a placeholder copy) of a media clip collection while preserving or maintaining the original media clip collection (and its media clips) unchanged. To illustrate, in one or more implementations, the media clip system 106 adds a control lens by adding one or more layers or elements of metadata to the media clip collection 2010, which modifies the appearance and / or access of the media clip collection. Further, in some implementations, each control lens is stored in a separate metadata file and, when presented, added to the media clip collection and / or media clips. Thus, a first set of viewers who are provided with a media clip collection with one or more control lenses applied will have a different experience than a second set of viewers (e.g., producers) who are provided with the same media clip collection without any control lenses, even though the media clip system 106 provides the same media clip collection to both sets of viewers.

[0300] As mentioned above, Figure 20B provides additional details regarding the various control lens elements shown in control lens element 2012 of Figure 20A. In particular, Figure 20B shows expanded options for appearance control lens 2020, access control lens 2022, and pop-up shop control lens 2024. For example, upon detecting selection of an access control lens element within shared element 2008 (or from another tool, menu, or command), media clip system 106 provides additional elements for appearance control lens 2020, as shown in Figure 20B.

[0301] As shown, the appearance control lens 2020 includes various exemplary options for modifying the appearance of a media clip collection. As shown, the media clip system 106 can apply visual filters. For example, the media clip system 106 can add a color filter or lens over media clips in the media clip collection, apply aesthetic changes that invert colors, change the color scheme (e.g., make them black and white), or add stickers, apply a red-eye reduction filter to correct red eye, and / or utilize machine learning models to modify, touch up, or transform one or more media clips. Indeed, in some implementations, visual appearance filters allow user accounts to set defined visual parameters for viewing media clips in a media clip collection.

[0302] Also, as shown, the appearance control lens 2020 includes an option for adding watermarks. For example, the media clip system 106 adds a watermark, label, copyright notice, or other text to media clips in a media clip collection. Additionally, the appearance control lens 2020 includes an option for filtering out one or more media clips based on tags. For example, the media clip system 106 utilizes a list of tags to determine which media clips to display and which to hide (or perform another action). In an exemplary implementation, the media clip system 106 compares the tag list with the content item metadata for each media clip to determine whether or how to display the media clip.

[0303] 20B also includes an access control lens 2022. As shown, the access control lens 2022 includes various exemplary options for modifying access rights to the media clip collection. For example, the access control lens 2022 includes an option for restricting downloads. For example, in one or more implementations, the media clip system 106 prevents, restricts, or otherwise controls downloading of media clips in the media clip collection. In some cases, the download control lens restricts the download or print resolution of the media clip.

[0304] As shown, the access control lens 2022 includes options for restricting access to the media clip collection. In some examples, the media clip system 106 restricts access to loading or viewing media clips. Indeed, the media clip system 106 can restrict access to the media clip collection in several ways (e.g., password protection). Also shown, the access control lens 2022 includes options for subscriptions and memberships. For example, the media clip system 106 prevents access or notifications when the media clip collection is updated unless the viewer has a subscription or membership. Indeed, the media clip system 106 can apply various control lenses that provide more complete control over the media clips in the media clip collection.

[0305] Additionally, FIG. 20B includes a pop-up shop control lens 2024. As shown, the pop-up shop control lens 2024 includes various exemplary options for setting up a temporary shop for a media clip collection. As shown, in some examples, the pop-up shop control lens 2024 includes a control lens for selecting an expiration option, such as removing the temporary shop upon the occurrence of an expiration action, such as the expiration of a timer or the occurrence of a large number of views. As another example, the pop-up shop control lens 2024 can include a control lens option for adding collaborators to the pop-up shop and / or can include another media clip collection, such as media clips from multiple user accounts that make up the media clip collection. Additionally, the media clip system 106 can add a control lens for adding prices or values ​​to media clips in a media clip collection as part of a temporary or more permanent e-commerce site centered around the media clip collection.

[0306] In various implementations, the media clip system 106 can apply multiple control lenses to a media clip collection. For example, the media clip system 106 applies one or more appearance control lenses 2020 and one or more access control lenses 2022 to a media clip collection. Additionally, the media clip system 106 can create different versions of a media clip collection that apply different control lenses.

[0307] To illustrate, Figure 20C shows a media clip collection 2010 and exemplary control lenses, including a subscription control lens 2030, a watermark control lens 2032, and a red-eye control lens 2034. Figure 20C also shows multiple viewers, including a first viewer 2040 (i.e., viewer A) and a second viewer 2042 (i.e., viewer B). As shown, the media clip system 106 can provide the same media clip collection to both the first viewer 2040 and the second viewer 2042.

[0308] Additionally, the media clip system 106 can apply different control lenses to the media clip collection 2010. As shown, the media clip system 106 applies a subscription control lens 2030, a watermark control lens 2032, and a red-eye control lens 2034 to the media clip collection before providing it to a first viewer 2040. Additionally, the media clip system 106 applies the red-eye control lens 2034 to the media clip collection 2010 before providing it to a second viewer 2042. Thus, the viewer at the second viewer 2042 can access the media clip collection 2010 with the red-eye filter applied without the additional restrictions imposed on the viewer at the first viewer 2040. Additionally, as described above, the media clip system 106 can provide different versions of the media clip collection 2010 to viewers without changing the content of the underlying media clips in the media clip collection. In an additional exemplary control lens combination, the media clip system 106 can provide a low-resolution version of the content in the collection to a first viewer (who does not pay) while providing a high-resolution version of the content in the collection to a second viewer (who is a paying subscriber). This can be true for both images and videos in the collection. Thus, the media clip system 106 uses the lens concept to efficiently and effectively share content with different audiences with different objectives.

[0309]

[0047] Referring now to the next figure, Figure 21 illustrates publishing and sharing a media clip collection in accordance with one or more embodiments. As shown, Figure 21 includes a media clip collection 2102. In various implementations, the media clip system 106 allows user accounts to publish, share, embed, and / or market media clips within the media clip collection.

[0310] To illustrate, in various implementations, the media clip system 106 allows a user account to publish a media clip collection 2102 as a public media clip collection 2104. In some implementations, the media clip system 106 publishes the media clip collection 2102 to an online media clip collection repository or website. In this manner, other user accounts can view, access, search, etc. the published media clip collection 2104. In example implementations, the media clip system 106 can publish the media clip collection 2102 with a selected layout configuration and / or one or more control lenses, as described above. In example implementations, the media clip system 106 generates a simple, readable, and understandable link (e.g., a URL) for accessing the media clip collection 2102 (e.g., dropbox.com / user_account_name / collection_name).

[0311] In some implementations, the media clip system 106 allows user accounts to market their media clip collections 2102. For example, in various implementations, the media clip system 106 allows user accounts to convert their media clip collections 2102 into media clip collection stores 2106 where visitors can purchase access to media clips, subscription prints, downloads, etc. As described above, the media clip system 106 can provide various control lenses to facilitate converting a media clip collection into a media clip collection store 2106.

[0312] Additionally, in one or more implementations, the media clip system 106 may enable user accounts to share their media clip collections 2102. For example, the media clip system 106 provides an option to convert a media clip collection into a media clip collection blog 2108, as shown. In various implementations, the media clip system 106 allows a media clip collection to be converted into a journal, a news article, a content gallery, or another shareable format. In an example implementation, the media clip system 106 allows a user account to share a link (e.g., a simple, readable, and understandable URL) to their media clip collection 2102 with another user account or computing device.

[0313] As also shown, in example implementations, the media clip system 106 can enable a user account to embed a media clip collection 2102 in one or more websites and / or applications. Furthermore, the embedded media clip collection 2110 can be a public media clip collection 2104, a media clip collection store 2106, or another type of shared media clip collection. In this manner, a user account can provide a media clip collection to various connected sites (e.g., websites). Furthermore, in these implementations, when media clips in the media clip collection 2102 change, the media clip system 106 can automatically propagate the changes to each of the embedded media clip collections (and / or other shared media clip collections), thus maintaining a consistent presentation of the media clip collection across different sites.

[0314] Just as user accounts can share media clips and media clip collections, in various implementations, user accounts can add media clips from other user accounts to their media clip library. To illustrate, Figures 22A-22B show adding attributed digital media clips and attributed media clip collections to a media clip library according to one or more embodiments. As shown, Figure 22A includes an application 2204 (e.g., a web browser) showing a second user media clip 2202b shared by a second user account (i.e., User 2).

[0315] As shown, the application 2204 also includes an add media clip element 2206 that enables a user account to add a second user media clip 2202b to a user account (e.g., a first user account). Upon detecting a selection of the add media clipping element 2206 by the user account, the media clipping system 106 can add the second user media clipping to the first user media clipping library 2210 of the user account, shown as a first user media clipping 2202a in the first user media clipping library 2210.

[0316] In various implementations, the media clip system 106 also determines an attribution connection 2208 when adding a media clip from one user account to another. In this way, the media clip system 106 can track the source and origin of a media clip even if the media clip is modified. In some implementations, the media clip system 106 maintains a media clip attribute table that maintains the attribute chain of the media clip (e.g., associated with the original media clip or stored in a central attribute database). The media clip attribute table includes the original source of the first user media clip 2102a (e.g., a second user), other user accounts that also have the second user media clip 2202b in their media clip library, and / or other user accounts that added the first user media clip 2102a from the first user account to their user accounts.

[0317] Indeed, in various implementations, the media clip system 106 can track the attribution of media clips as they circulate among various media clip collections and user accounts. By maintaining attribution connections 2208, the media clip system 106 can not only track where media clips have been shared, but also use the shared media clips to identify and suggest additional related media clips for addition to the user account's media clip library, as described in detail above.

[0318] In addition to adding individual media clips, in various implementations, the media clip system 106 facilitates adding another user's entire media clip collection to a user account. To illustrate, FIG. 22B shows a user account adding an external media clip collection to the user account. In particular, FIG. 22B shows a second user account's original media clip collection 2220b with original media clips 2226b. For example, the second user account publishes the original media clip collection 2220b to a media clip collection sharing website or repository.

[0319] As shown, the original media clip collection 2220b includes a user account customization or remix media clip collection element 2222. In various implementations, the remix media clip collection element 2222 allows a user account to add the original media clip collection 2220b to the user account, as well as customize or redesign the media clip collection as needed.

[0320] To illustrate, FIG. 22B shows a remixed media clip collection 2220a for a first user account. In one or more embodiments, the option to remix the collection is based on user permission. As shown, the remixed media clip collection 2220a includes remixed media clips 2226a that match original media clips 2226b but are organized differently. For example, the media clip system 106 allows the first user account to remix the media clip collection by organizing the remixed media clips 2226a in a different order, configuration, etc. Additionally, in some implementations, the user account can delete one or more of the remixed media clips 2226a from the remixed media clip collection 2220a.

[0321] Additionally, FIG. 22B includes attribution connections 2224. Just as a single media clip can be considered an original creation, a media clip collection can also be considered an original creation, even if the media clip collection is remixed. Indeed, in various implementations, the media clip system 106 maintains one or more attribution connections to track how media clips and media clip collections are shared between user accounts. Furthermore, the media clip system 106 can ensure that proper credit is given to the creator or creator of a media clip collection, even if the media clips within the media clip collection are modified or customized in another user account. In other words, the media clip system 106 maintains attribution metadata at both the collection and media clip levels, which allows attribution of the work to be maintained throughout various uses and operations of the collection.

[0322] The foregoing figures, corresponding text, and examples provide several different systems, methods, techniques, components, and / or devices of the digital media clip system 106 according to one or more embodiments. In addition to the above description, one or more embodiments may also be described in terms of flowcharts that include operations for achieving a particular result. For example, FIG. 23 shows a flowchart corresponding to one or more implementations of the media clip system 106 described above. Furthermore, the operations outlined in FIG. 23 are provided only as examples, and some of the operations may be optional, combined into fewer operations, or expanded into additional operations without detracting from the essence of the disclosed embodiments. Furthermore, operations described herein may be repeated or performed in parallel with each other or with different instances of the same or similar operations.

[0323] Furthermore, the digital media clip system 106 may perform one or more operations in a sequence of operations in addition to, or instead of, one or more operations described in connection with other figures. While FIG. 23 illustrates operations according to one embodiment, alternative embodiments may omit, add, rearrange, and / or modify any of the illustrated operations. Furthermore, each of the operations in FIG. 23 may be performed as part of a method. Alternatively, a non-transitory computer-readable medium may include instructions that, when executed by one or more processors, cause a computing device to perform the operations of FIG. 23. In some embodiments, a system may perform the operations of FIG. 23.

[0324] To illustrate, Figure 23 includes a series of operations 2300 for providing one or more control lenses to a media clip collection in accordance with one or more embodiments. As shown, the series of operations 2300 includes an operation 2310 for generating a media clip collection. For example, operation 2310 can include generating a media clip collection that displays a plurality of digital media clips, each having a content item, content metadata, and contextual metadata.

[0325] In one or more embodiments, operation 2310 generates a media clip collection by automatically grouping related digital media clips from the plurality of digital media clips based on analyzing content metadata and contextual metadata of the plurality of digital media clips within the plurality of digital media clips. In some implementations, operation 2310 includes generating a set of combined digital media clips including a first digital media clip combined with a second digital media clip in a defined presentation order, wherein the first digital media clip cannot be provided for display in the media clip collection without at least some of the second digital media clips being provided for display. In one or more implementations, operation 2310 also includes generating a digital canvas displaying the digital media clips of the plurality of digital media clips, the file, and the combined set of digital media clips, wherein the digital media clips are stored external to the digital canvas.

[0326] In some implementations, operation 2310 includes providing an interactive graphical user interface within the digital canvas that displays a set of digital media clips from the plurality of digital media clips and digital canvas metadata generated from the set of digital media clips. In various implementations, operation 2310 includes detecting the addition of a new digital media clip moved to an edge of the digital canvas and dynamically expanding the size of the digital canvas to encompass the new digital media clip based on the addition of the new digital media clip.

[0327] Further, the series of operations 2300 includes an operation 2320 for identifying a control lens that modifies the appearance of or access to the media clip collection. For example, operation 2320 can include identifying a plurality of control lenses that modify the appearance of or access to the media clip collection without modifying the plurality of digital media clips. In one or more implementations, operation 2320 can include identifying a plurality of control lenses that modify the appearance of or access to the media clip collection without modifying the plurality of digital media clips.

[0328] The series of operations 2300 also includes an operation 2330 of providing a first control lens to the media clip collection. For example, operation 2320 may include providing a first control lens of the plurality of control lenses to the media clip collection. In one or more implementations, operation 2330 may include modifying the appearance of the plurality of digital media clips in the media clip collection by applying the first control lens to the plurality of digital media clips.

[0329] In some implementations, operation 2330 can include modifying access to the plurality of digital media clips and the media clip collection by applying a first control lens to the media clip collection. In various implementations, operation 2330 can include providing the media clip collection to a first set of computing devices with the first control lens applied to the media clip collection. In an example implementation, operation 2330 can include providing a first control lens of the plurality of control lenses to the media clip collection, the first control lens modifying the appearance of the plurality of digital media clips in the media clip collection.

[0330] Further, the series of operations 2300 includes an operation 2340 providing a second control lens to the media clip collection. For example, operation 2340 can include providing a second control lens of a plurality of control lenses to the media clip collection, where the second control lens is different from the first control lens. In various implementations, operation 2340 can include providing the media clip collection to a second set of computing devices with the second control lens applied to the media clip collection. In some implementations, operation 2340 can include providing the media clip collection for display arranged in one of a plurality of layouts, including a grid layout, a gallery layout, a timeline layout, and a freeform layout. In one or more implementations, operation 2340 can include providing the media clip collection with a second control lens that modifies viewing access to the media clip collection. In an example implementation, the second control lens includes access control for viewing, sharing, copying, or printing the plurality of digital media clips in the media clip collection.

[0331] 23. The series of operations 2300 may also include operations not included in FIG. 23. To illustrate, in various implementations, the series of operations 2300 includes the additional operations of adding a shared digital media clip from an external shared media clip collection to the media clip collection, providing the shared digital media clip for display in the media clip collection with multiple digital media clips, and providing, based on receiving a selection indication of the shared digital media clip, an attribute table for display including multiple external media clip collections including the shared digital media clip and the source media clip collection of the shared digital media clip. In some implementations, the series of operations 2300 includes the additional operations of identifying a digital media clip within the digital canvas, determining additional content metadata for the digital media clip based on the content metadata and contextual metadata of the digital media clip, and providing the additional content metadata for display in association with the digital media clip. Components of the Media Clip System

[0332] Embodiments of the present disclosure may comprise or utilize special purpose or general purpose computers including computer hardware such as, for example, one or more processors and system memory, as described in more detail below. Embodiments within the scope of the present disclosure also include physical and other computer-readable media for carrying or storing computer-executable instructions and / or data structures. In particular, one or more of the processes described herein may be embodied, at least in part, in a non-transitory computer-readable medium and implemented as instructions executable by one or more computing devices (e.g., any of the media content access devices described herein). Generally, a processor (e.g., a microprocessor) receives instructions from a non-transitory computer-readable medium (e.g., memory) and executes those instructions, thereby performing one or more processes, including one or more of the processes described herein.

[0333] Computer-readable media may be any available media that can be accessed by a general-purpose or special-purpose computer system. Computer-readable media that store computer-executable instructions are non-transitory computer-readable storage media (devices). Computer-readable media that carry computer-executable instructions are transmission media. Thus, by way of example, and not limitation, embodiments of the present disclosure may include at least two distinctly different types of computer-readable media: non-transitory computer-readable storage media (devices) and transmission media.

[0334] Non-transitory computer-readable storage media (devices) include RAM, ROM, EEPROM, CD-ROM, solid-state drives ("SSD") (e.g., RAM-based), flash memory, phase-change memory ("PCM"), other types of memory, other optical disk storage, magnetic disk storage, or other magnetic storage devices, or any other medium that can be used to store computer-executable instructions or program code means necessary to form data structures and that can be accessed by a general-purpose or special-purpose computer.

[0335] A "network" is defined as one or more data links that enable the transfer of electronic data between computer systems and / or modules and / or other electronic devices. When information is transferred or provided to a computer over a network or another communications connection (wired, wireless, or a combination of wired or wireless), the computer properly views the connection as a transmission medium. Transmission media can include network links and / or data links that can be used to carry computer-executable instructions or program code means necessary to form data structures and that can be accessed by a general-purpose or special-purpose computer. Combinations of the above should also be included within the scope of computer-readable media.

[0336] Furthermore, upon reaching various computer system components, program code means in the form of computer-executable instructions or data structures may be automatically transferred from transmission media to non-transitory computer-readable storage media (devices) (or vice versa). For example, computer-executable instructions or data structures received over a network or data link may be buffered in RAM within a network interface module (e.g., a "network interface card") and then eventually transferred to computer system RAM and / or less volatile computer storage media (devices) in the computer system. It should therefore be understood that non-transitory computer-readable storage media (devices) may be included in computer system components that also (or primarily) utilize transmission media.

[0337] Computer-executable instructions include, for example, instructions and data that, when executed by a processor, cause a general-purpose computer, a special-purpose computer, or a special-purpose processing device to perform a particular function or group of functions. In some embodiments, computer-executable instructions are executed by a general-purpose computer to transform the general-purpose computer into a special-purpose computer that implements elements of the present disclosure. Computer-executable instructions may be, for example, binaries, intermediate format instructions such as assembly language, or source code. While the subject matter has been described in language specific to structural features and / or methodological acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the features or acts described above. Rather, the described features and acts are disclosed as exemplary forms of implementing the claims.

[0338] Those skilled in the art will appreciate that the present disclosure may be practiced in networked computing environments having many types of computer system configurations, including personal computers, desktop computers, laptop computers, message processors, handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, mobile phones, PDAs, tablets, pagers, routers, switches, etc. The present disclosure may also be practiced in distributed system environments where tasks are performed by both local and remote computer systems that are linked through a network (either by wired data links, wireless data links, or a combination of wired and wireless data links). In a distributed system environment, program modules may be located in both local and remote memory storage devices.

[0339] Embodiments of the present disclosure may also be implemented in a cloud computing environment. As used herein, the term "cloud computing" refers to a model for enabling on-demand network access to a shared pool of configurable computing resources. For example, cloud computing may be used in markets to provide ubiquitous, convenient, on-demand access to a shared pool of configurable computing resources. The shared pool of configurable computing resources may be rapidly provisioned through virtualization, released with low management effort or service provider interaction, and then scaled accordingly.

[0340] Cloud computing models may consist of various characteristics, such as, for example, on-demand self-service, broad network access, resource pooling, rapid elasticity, and metered services. Cloud computing models may also expose various service models, such as, for example, Software as a Service ("SaaS"), Platform as a Service ("PaaS"), and Infrastructure as a Service ("IaaS"). Cloud computing models may also be deployed using different deployment models, such as private cloud, community cloud, public cloud, and hybrid cloud. Also, as used herein, "cloud computing environment" refers to an environment in which cloud computing is utilized.

[0341] FIG. 24 illustrates a block diagram of a computing device 2400 that may be configured to perform one or more of the processes described above. It will be understood that the client devices and / or server devices described herein may comprise one or more computing devices, such as client device 102 and / or client device 400. As shown in FIG. 24 , computing device 2400 may comprise a processor 2402, a memory 2404, a storage device 2406, an I / O interface 2408, and a communication interface 2410 that may be communicatively coupled by a communication infrastructure 2412. Although computing device 2400 is illustrated in FIG. 24 , the components illustrated in FIG. 24 are not intended to be limiting. In other embodiments, additional or alternative components may be used. Moreover, in particular embodiments, computing device 2400 may include fewer components than those illustrated in FIG. 24 . The components of computing device 2400 illustrated in FIG. 24 will now be described in further detail.

[0342] In particular embodiments, processor 2402 includes hardware for executing instructions, such as those making up a computer program. By way of example and not limitation, to execute instructions, processor 2402 may retrieve (or fetch) instructions from an internal register, an internal cache, memory 2404, or storage device 2406, decode them, and execute them. In particular embodiments, processor 2402 may include one or more internal caches for data, instructions, or destinations. By way of example and not limitation, processor 2402 may include one or more instruction caches, one or more data caches, and one or more translation lookaside buffers (TLBs). Instructions in an instruction cache may be copies of instructions in memory 2404 or storage device 2406. Additionally, processor 2402 may include or represent one or more processors or processor cores.

[0343] The memory 2404 may be used to store data, metadata, and programs for execution by the processor. The memory 2404 may include one or more of volatile and non-volatile memory, such as random access memory (“RAM”), read-only memory (“ROM”), solid-state disk (“SSD”), flash, phase-change memory (“PCM”), or other types of data storage. The memory 2404 may be internal or distributed memory.

[0344] The storage device 2406 includes storage devices for storing data or instructions. By way of example and not limitation, the storage device 2406 may comprise the non-transitory storage media described above. The storage device 2406 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. The storage device 2406 may include removable or non-removable (or fixed) media, as appropriate. The storage device 2406 may be internal or external to the computing device 2400. In particular embodiments, the storage device 2406 is a non-volatile solid-state memory. In other embodiments, the storage device 2406 includes read-only memory (ROM). Where appropriate, this ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically alterable ROM (EAROM), or a flash memory, or a combination of two or more of these.

[0345] The I / O interface 2408 enables a user to provide input to, receive output from, or otherwise transfer data to or receive data from the computing device 2400. The I / O interface 2408 may include a mouse, a keypad or keyboard, a touchscreen, a camera, an optical scanner, a network interface, a modem, other known I / O devices, or a combination of such I / O interfaces. The I / O interface 2408 may include one or more devices for presenting output to a user, including, but not limited to, a graphics engine, a display (e.g., a display screen), one or more output drivers (e.g., a display driver), one or more audio speakers, and one or more audio drivers. In particular embodiments, the I / O interface 2408 is configured to provide graphical data to a display for presentation to a user. The graphical data may represent one or more graphical user interfaces and / or any other graphical content and may fulfill a particular implementation.

[0346] The communications interface 2410 may include hardware, software, or both. In any event, the communications interface 2410 may provide one or more interfaces for communications (e.g., packet-based communications, etc.) between the computing device 2400 and one or more other computing devices or networks. By way of example and not limitation, the communications interface 2410 may include a network interface controller (NIC) or network adapter for communicating with an Ethernet or other wired-based network, or a wireless NIC (WNIC) or wireless adapter for communicating with a wireless network such as WI-FI.

[0347] Additionally or alternatively, communication interface 2410 may facilitate communication with one or more portions of an ad-hoc network, a personal area network (PAN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), or the Internet, or a combination of two or more of these. One or more portions of one or more of these networks may be wired or wireless. By way of example, communication interface 2410 may enable communication with a wireless PAN (WPAN) (e.g., a BLUETOOTH WPAN, etc.), a Wi-Fi network, a Wi-Max network, a cellular telephone network (e.g., a Global System for Mobile Communications (GSM) network, etc.), or other suitable wireless network, or a combination thereof.

[0348] Additionally, communication interface 2410 may facilitate communications using various communication protocols. Examples of secure communication protocols that may be used include data transmission media, Transmission Control Protocol ("TCP"), Internet Protocol ("FTP"), Hypertext Transfer Protocol ("HTTP"), Session Initiation Protocol ("SIP"), Simple Object Access Protocol ("SOAP"), Extensible Markup Language ("XML"), Simple Mail Transfer Protocol ("RTP"), Real-Time Transport Protocol ("UDP"), Global System for Mobile Communications ("GSM") technology, Code Division Multiple Access ("CDMA") technology, Time Division Multiple Access ("TDMA") technology, Short Message Service ("SMS"), Multimedia Message Service ("MMS"), Radio Frequency ("RF") signaling technologies, including but not limited to Long Term Evolution (LTE) technology, wireless communication technologies, in-band and out-of-band signaling technologies, and other suitable communication networks and technologies.

[0349] Communications infrastructure 2412 may include hardware, software, or both that couple together components of computing device 2400. By way of example and not limitation, communications infrastructure 2412 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infiniband interconnect, a Low Pin Count (LPC) bus, a memory bus, a MicroChannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCIe) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standard Related Local (VLB) bus, or another suitable bus, or a combination thereof.

[0350] 25 is a schematic diagram illustrating an environment 2500 in which one or more embodiments of content management system 110 can be implemented. For example, as shown in FIG. 25, content management system 110 can generate, store, manage, receive, and transmit digital content (such as digital documents and other content items). For example, content management system 110 can send and receive digital content to and from client devices 2506 over network 2504. In particular, content management system 110 can store and manage collections of digital content. Content management system 110 can manage the sharing of digital content among computing devices associated with multiple users (i.e., user accounts in content management system 110). For example, content management system 110 can facilitate a user sharing digital content with another user of content management system 110.

[0351] In particular, content management system 110 can manage the synchronization of digital content across multiple client devices associated with one or more users. For example, a user can edit digital content using client device 2506. Content management system 110 can cause client device 2506 to send the edited digital content to content management system 110. Content management system 110 then synchronizes the edited digital content on one or more additional computing devices.

[0352] In addition to synchronizing digital content across multiple devices, one or more embodiments of content management system 110 can provide efficient storage options for users with large collections of digital content. For example, content management system 110 can store a collection of digital content on content management system 110, while client device 2506 stores only reduced-size versions of the digital content. A user can navigate and view reduced-size versions of the digital content (e.g., thumbnails of digital images) on client device 2506. In particular, one way a user can experience digital content is by viewing reduced-size versions of the digital content on client device 2506.

[0353] Another way a user can experience digital content is by selecting a scaled-down version of the digital content and then requesting a full-size or high-size version of the digital content from content management system 110. In particular, when a user selects a scaled-down version of the digital content, client device 2506 sends a request to content management system 110 requesting digital content related to the scaled-down version of the digital content. Content management system 110 can respond to the request by sending the digital content to client device 2506. Client device 2506 can then present the digital content to the user upon receiving it. In this way, a user can access a large collection of digital content while minimizing the number of resources used on client device 2506.

[0354] Client device 2506 may be a desktop computer, a laptop computer, a tablet computer, a personal digital assistant (PDA), an in- or out-of-car navigation system, a smart TV, a virtual reality (VR) or augmented reality (AR) device, a handheld device, a wearable device, a smartphone or other cellular or mobile phone, or a mobile gaming device, another mobile device, or other suitable computing device. Client device 2506 may run one or more client applications, such as a web browser (e.g., Microsoft Windows Internet Explorer, Mozilla Firefox, Apple Safari, Google Chrome, Opera, etc.) or a native or dedicated client application (e.g., Facebook for iPhone or iPad, Facebook for Android, etc.), to access and view content over network 2504.

[0355] Network 2504 may represent a network (such as the Internet, a corporate intranet, a virtual private network (VPN), a local area network (LAN), a wireless local area network (WLAN), a cellular network, a wide area network (WAN), a metropolitan area network (MAN), or a combination of two or more such networks) or a collection of networks through which client device 2506 can access content management system 110.

[0356] In the foregoing specification, the invention has been described with reference to specific exemplary embodiments thereof. Various embodiments and aspects of the invention are described with reference to the details discussed herein, and the accompanying drawings illustrate various embodiments. The above description and drawings are illustrative of the invention and should not be construed as limiting the invention. Numerous specific details are set forth in order to provide a thorough understanding of various embodiments of the invention.

[0357] The present invention may be embodied in other specific forms without departing from the spirit and essential characteristics thereof. The described embodiments are to be considered in all respects as illustrative only and not restrictive. For example, methods described herein may be performed with fewer or more steps / actions, or the steps / actions may be performed in a different order. In addition, steps / actions described herein may be repeated or performed in parallel or concurrently with each other for different instances of the same or similar steps / actions. The scope of the present invention is therefore indicated by the appended claims, rather than by the foregoing description. All changes that come within the meaning and range of equivalency of the claims are to be embraced within their scope.

Claims

1. 1. A computer-implemented method comprising: maintaining, for a user account in the content management system, a media clip library including digital media clips each having content items, content metadata, and contextual metadata; detecting content items not associated with the user account; determining a correlation between the content item not associated with the user account and one or more digital media clips in the media clip library; generating a new digital media clip from the content item not associated with the user account based on the correlation, thereby adding the new digital media clip to the media clip library; 11. A computer-implemented method comprising:

2. The computer-implemented method of claim 1, wherein determining the correlation between the content item not associated with the user account and the one or more digital media clips in the media clip library includes comparing the content item not associated with the user account with contextual metadata of digital media clips in the media clip library.

3. adding the new digital media clip to the media clip library comprises: providing recommendations for generating digital media clips of the content items not associated with the user account; adding the digital media clip to the media clip library based on detecting an interaction that confirms the recommendation; The computer-implemented method of claim 1 , comprising:

4. Adding the new digital media clip to the media clip library, comprising: identifying a plurality of content items not associated with the user account; selecting a content item from the plurality of content items by comparing the plurality of content items with contextual metadata of digital media clips in the media clip library; providing a suggestion to add the selected content item to the media clip library as a digital media clip; The computer-implemented method of claim 1 , comprising:

5. For a digital media clip in the media clip library of the user account, determining that a copy of the digital media clip is already located in a second media clip library of a second user account on the content management system; determining additional correlations between additional digital media clips in the second media clip library of the second user account and the digital media clips in the media clip library of the user account; providing content items from the additional digital media clips in the second media clip library of the second user account based on the additional correlation; and The computer-implemented method of claim 1 , further comprising:

6. The method of claim 1, further comprising: generating the new digital media clip from the content item that is not associated with the user account; capturing the content item not associated with the user account; extracting content metadata from the content items that are not associated with the user account; generating contextual metadata from the contextual signal corresponding to one or more digital environments in which the content items not associated with the user account currently reside; The computer-implemented method of claim 1 , further comprising:

7. identifying a plurality of content items not associated with the user account based on a search query; determining one or more implicit user preferences based on contextual metadata of digital media clips in the media clip library; filtering the plurality of content items based on the one or more implicit user preferences to identify a subset of content items; providing the subset of content items in response to the search query; The computer-implemented method of claim 1 , further comprising:

8. identifying a target digital media clip; receiving a user input indicating a level of relevance ranging from highly correlated to no correlation; tuning parameters of a media clip classification machine learning model based on the relevance levels; identifying one or more content items utilizing the tuned media clip classification machine learning model and the target digital media clip; and The computer-implemented method of claim 1 , further comprising:

9. When executed by at least one processor, the computer device maintaining, for a user account in the content management system, a media clip library including digital media clips each having content items, content metadata, and contextual metadata; detecting content items not associated with the user account; determining a correlation score between the content item not associated with the user account and one or more digital media clips in the media clip library; providing recommendations for generating digital media clips for the content items not associated with the user account in the media clip library based on the correlation scores; generating a digital media clip of the content item not associated with the user account based on receiving a user confirmation of the recommendation; A non-transitory computer-readable storage medium containing instructions to cause

10. When executed by the at least one processor, the computing device: identifying a plurality of content items not associated with the user account, the plurality of content items including the content item not associated with the user account; selecting a content item from the plurality of content items based on the correlation score; 10. The non-transitory computer-readable storage medium of claim 9, further comprising instructions to:

11. 10. The non-transitory computer-readable storage medium of claim 9, further comprising instructions that, when executed by the at least one processor, cause the computing device to add the digital media clip to the media clip library of the user account.

12. When executed by the at least one processor, the computing device: determining a correlation between contextual metadata of the digital media clip and contextual metadata from a second media clip library of a second user account of the content management system; providing an additional recommendation to the second user account for adding the digital media clip to a second media clip library associated with the second user account based on the correlation; and 10. The non-transitory computer-readable storage medium of claim 9, further comprising instructions to:

13. 10. The non-transitory computer-readable storage medium of claim 9, further comprising instructions that, when executed by the at least one processor, cause the computing device to determine the correlation score between the content item not associated with the user account and the one or more digital media clips in the media clip library by comparing contextual metadata associated with the content item not associated with the user account with contextual metadata of digital media clips in the media clip library.

14. 10. The non-transitory computer-readable storage medium of claim 9, further comprising instructions that, when executed by the at least one processor, cause the computing device to select a content item based on a correlation score that meets a digital media clip suggestion threshold.

15. 1. A system comprising: at least one processor; When executed by the at least one processor, the system: maintaining, for a user account in the content management system, a media clip library including digital media clips each having content items, content metadata, and contextual metadata; identifying a plurality of content items not associated with the user account; determining, from the plurality of content items, one or more content items that correlate to the contextual metadata of one or more digital media clips in the media clip library by comparing the plurality of content items with contextual metadata of one or more digital media clips in the media clip library; adding one or more digital media clips to the media clip library by generating one or more digital media clips from the one or more content items that correlate to the contextual metadata of one or more digital media clips in the media clip library; a non-transitory computer memory containing instructions to cause A system comprising:

16. 16. The system of claim 15, further comprising instructions that, when executed by the at least one processor, cause the system to provide one or more recommendations for generating the one or more digital media clips of the one or more content items.

17. 16. The system of claim 15, further comprising instructions that, when executed by the at least one processor, cause the system to identify the plurality of content items from additional media clip libraries of additional users of the content management system.

18. When executed by the at least one processor, the system: identifying a set of content items not associated with the user account based on a search query; determining one or more implicit user preferences based on contextual metadata of digital media clips in the media clip library; filtering the set of content items based on the one or more implicit user preferences to identify a subset of content items; providing the subset of content items in response to the search query; The system of claim 15 further comprising instructions to:

19. When executed by the at least one processor, the system: identifying a target digital media clip; receiving a user input indicating a level of relevance ranging from highly correlated to no correlation; tuning parameters of a media clip classification machine learning model based on the relevance levels; identifying a set of content items utilizing the tuned media clip classification machine learning model and the target digital media clip; The system of claim 15 further comprising instructions to:

20. When executed by the at least one processor, the system: Capturing content items from a digital environment; extracting content metadata from the content items from the digital environment; generating contextual metadata from contextual signals detected from the digital environment; 16. The system of claim 15, further comprising instructions to generate a digital media clip of the content item from the digital environment by

Images