Automatic media asset suggestions for presentation of selected user media items

Abstract

The present disclosure relates to automatic media asset recommendations for presentation of selected user media items. Provided are techniques for recommending media assets that include requesting a set of candidate media assets for a set of user media items based on a knowledge graph metadata network that describes the set of user media items, receiving metadata for the set of candidate media assets, determining one or more ranked sets of media assets based on the received metadata, and outputting the determined one or more ranked sets of media assets.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Application No. 63 / 195,562, filed on June 1, 2021, which is hereby incorporated by reference. Technical Field

[0003] Embodiments described herein relate to automatic media asset recommendations. More specifically, embodiments described herein relate to organizing, describing, and / or retrieving media assets based on contextual analysis of a user's media items so that the media assets can be presented to a user of a computing system in the form of recommendations for selection for playback to accompany a multimedia presentation of the user's collection of media items. Background Art

[0004] Modern consumer electronics have enabled users to create, store, and access a large number of user media items. For example, a computing system (e.g., a smartphone, a stationary computer system, a portable computer system, a media player, a tablet computer system, a wearable computer system or device, etc.) can create, store, or access a collection of user media items (also referred to as a user media item collection) that includes hundreds or thousands of user media items (e.g., images, videos, music, etc.).

[0005] Managing a user media item collection can be a resource-intensive endeavor for a user. For example, retrieving multiple user media items representing significant moments or events in a user's life from a relatively large user media item collection may require the user to sift through many unrelated user media items. This process can be difficult and frustrating for many users. A digital user media item management system can assist in managing a user media item collection. A user media item management system represents an interwoven system that combines software, hardware, and / or other services to manage, store, ingest, organize, and retrieve user media items from a user media item collection.

[0006] In addition to the aforementioned difficulties that users can face when managing large collections of user media items (e.g., locating and / or retrieving multiple user media items that represent important moments, events, people, locations, themes, or themes in a user’s life), users can also have difficulty determining (or can not take the time to determine) which user media items are meaningful to view (e.g., in the form of a multimedia presentation (such as a slideshow), also referred to herein as a “memory”) and / or share with third parties (e.g., other users of a similar user media management system and / or the user’s social contacts). Users can also not want to take the time to determine suitable sequences, clusters, layouts, themes, transition types and durations, etc. of user media items to build such multimedia presentations of user media items. Furthermore, users can have difficulty determining suitable media assets (e.g., one or more songs to use as “soundtracks”) to associate with playback of a set of user media items to be included in a multimedia presentation. Accordingly, there is a need for methods, apparatuses, computer-readable media, and systems to provide more intelligent and automated audio media asset recommendations for multimedia presentations of sets of user media items, e.g., based on one or more features of user media items in the set of user media items. SUMMARY

[0007] Disclosed herein are techniques for recommending media assets, including a method comprising: requesting a set of candidate media assets for a set of user media items based on a network of knowledge graph metadata describing the set of user media items; receiving metadata for the set of candidate media assets; determining one or more ranked sets of media assets based on the received metadata; and outputting the determined one or more ranked sets of media assets.

[0008] According to aspects of the disclosure, an electronic device is disclosed, comprising: a memory; and one or more processors operatively coupled to the memory, wherein the one or more processors are configured to execute instructions that cause the one or more processors to: request a set of candidate media assets for a set of user media items based on a network of knowledge graph metadata describing the set of user media items; receive metadata for the set of candidate media assets; determine one or more ranked sets of media assets based on the received metadata; and output the determined one or more ranked sets of media assets.

[0009] Another aspect relates to a non-transitory computer-readable medium storing instructions that, when executed, cause a processor to: request a set of candidate media assets for a set of user media items based on a network of knowledge graph metadata describing the set of user media items; receive metadata for the set of candidate media assets; determine one or more ranked sets of media assets based on the received metadata; and output the determined one or more ranked sets of media assets.

[0010] Other features or advantages of the embodiments described herein can be apparent from the accompanying drawings and from the detailed description which follows. BRIEF DESCRIPTION OF DRAWINGS

[0011] The embodiments described herein are illustrated by way of example and not limitation in the figures of the accompanying drawings in which like references indicate similar features. Further, in the figures, some conventional details are omitted in order to more clearly disclose the inventive concept described herein.

[0012] Figure 1 An asset management processing system including electronic components for performing media item management is shown in block diagram form in accordance with one embodiment.

[0013] Figure 2 An example of a time view user interface for presenting a collection of user media items based on a time at which the user media items were captured is shown in accordance with one embodiment.

[0014] Figure 3 An exemplary knowledge graph metadata network in accordance with aspects of the present disclosure is shown in block diagram form.

[0015] Figure 4 is a flowchart showing techniques for recommending media assets for memory playback in accordance with aspects of the present disclosure.

[0016] Figure 5 is a block diagram showing optional aspects for requesting candidate media assets in accordance with aspects of the present disclosure.

[0017] Figure 6 is a block diagram showing optional aspects for determining one or more ranked media asset sets in accordance with aspects of the present disclosure.

[0018] Figure 7 An exemplary user interface for displaying recommended media assets in accordance with aspects of the present disclosure is shown.

[0019] Figure 8 is a simplified functional block diagram of an exemplary programmable electronic device for performing user media item management in accordance with one embodiment. DETAILED DESCRIPTION

[0020] Figure 1A media access system 100 including a client device 101 and a server device 150 is shown in block diagram form. The client device 101 includes electronic components for performing user media item management in accordance with one or more embodiments described in this disclosure. The client device 101 can be housed in a single computing system such as a desktop computer system, a laptop computer system, a tablet computer system, a server computer system, a mobile phone, a media player, a personal digital assistant, a personal communicator, a gaming device, a network router or hub, a wireless access point (AP) or repeater, a set-top box, or a combination thereof. The components in the client device 101 can be spatially separated and implemented on separate computing systems connected by a communication technology 110, as described in further detail below.

[0021] For one embodiment, the client device 101 can include a processing unit 104, a memory 110, a user media item capture device 102, a sensor 122, and a peripheral device 118. For one embodiment, one or more components in the client device 101 can be implemented as one or more integrated circuits (ICs). For example, at least one of the processing unit 104, the communication technology 120, the user media item capture device 102, the peripheral device 118, the sensor 122, or the memory 110 can be implemented as a system-on-a-chip (SoC) IC, a three-dimensional (3D) IC, any other known IC, or a combination of any known ICs. For another embodiment, two or more components in the client device 101 are implemented together as one or more ICs. For example, at least two of the processing unit 104, the communication technology 120, the user media item capture device 102, the peripheral device 118, the sensor 122, or the memory 110 can be implemented together as a SoC IC. Each component of the client device 101 is described below.

[0022] As Figure 1As shown, the client device 101 may include a processing unit 104, such as a CPU, GPU, other integrated circuits (ICs), memory, and / or other electronic circuitry. For one embodiment, the processing unit 104 manipulates and / or processes user media item metadata associated with the user media items 112 or optional data 116 associated with the user media items (e.g., data objects, such as nodes, reflecting one or more people, places, points of interest, scenes, meanings, and / or events associated with a given user media item). The processing unit 104 may include a digital asset management system 106 for performing one or more embodiments of user media item management, as described herein. In some cases, the user media item management system 106 may include a candidate media asset ranking module 124 that ranks the received candidate media asset metadata 126. For one embodiment, the user media item management system 106 is implemented as hardware (e.g., electronic circuitry, circuitry, dedicated logic components, etc. associated with the processing unit 104), software (e.g., one or more instructions associated with a computer program executed by the processing unit 104, software running on a general-purpose computer system or a dedicated machine, etc.), or a combination thereof.

[0023] The user media item management system 106 can enable the client device 101 to generate and use a knowledge graph metadata network (also referred to more simply herein as a "knowledge graph" or "metadata network") 114 of user media metadata 112 as a multidimensional network. Metadata networks and multidimensional networks that can be used to implement the various techniques described herein are described in further detail, for example, in U.S. non-provisional patent application Ser. No. 15 / 391,269, filed on December 27, 2016, entitled "Notable Moments in a Collection of Digital Assets" (the "'269 application"), which is incorporated herein by reference. Figure 3 (which is described below) provides additional details regarding an exemplary metadata network 114.

[0024] In one embodiment, the user media item management system 106 may perform one or more of the following operations: (i) generate a metadata network 114; (ii) associate and / or present at least two user media items based on the metadata network 114, for example, as part of a moment; (iii) determine user media of interest in the user media collection based on the metadata network 114 and one or more other criteria and / or present the user media of interest to the user; and (iv) determine a ranked set of media assets from a media asset server based on information from the metadata network 114 and contextual analysis and / or present the ranked set of media assets to the user. As an example, a metadata network may be generated based on the user media items, and based on the metadata network, a presentation of a subset of the user media items may be automatically generated, and media assets may be automatically recommended to accompany the presentation.

[0025] The user media item management system 106 can obtain or receive a collection of user media item metadata 112 associated with a collection of user media items. As used herein, "media item" and its variations refer to data that can be stored in or as a digital form (e.g., a digital file, etc.). This digitized data includes, but is not limited to, the following: image media (e.g., still or animated images, etc.); audio media (e.g., songs, etc.); text media (e.g., e-books, etc.); video media (e.g., movies, etc.); and tactile media (e.g., vibration or motion provided in conjunction with other media, etc.). The above examples of digitized data can be combined to form multimedia (e.g., computer-animated cartoons, video games, etc.). A single user media item refers to a single instance of digitized data (e.g., an image, a song, a movie, etc.). Multiple user media items or a group of user media items refers to multiple instances of digitized data (e.g., multiple images, multiple songs, multiple movies, etc.). For the purposes of this disclosure, the use of a media item refers to one or more media items, including a single media item and a group of media items. For simplicity, the concepts described in this document use the operational examples of user media items as one or more images. It should be understood that user media items are not limited to these, and the concepts described in this document are applicable to other user media items (e.g., the different media items described above, etc.). It is worth noting that user media items include media items created by the user or media items created by the user and acquired by the user. In contrast, media assets may refer to media items that may (or have) been acquired by the user from a media store or other commercial entity (or shared with the user).

[0026] As used herein, "user media item collection" and variations thereof refer to a plurality of user media items that can be stored in one or more storage locations. As known, the one or more storage locations can be separated spatially or logically.

[0027] As used herein, "metadata," "user media item metadata," "user media item metadata," and variations thereof collectively refer to information about one or more user media items. Metadata can be: (i) a single instance of information about digitized data (e.g., a timestamp associated with one or more images, etc.); or (ii) a metadata grouping, which refers to a group consisting of multiple instances of information about digitized data (e.g., several timestamps associated with one or more images, etc.). There can also be many different types of metadata associated with a collection of user media items. Each type of metadata (also referred to as a "metadata type") describes one or more characteristics or attributes associated with one or more user media items. More details about the various types of metadata that can be stored in a collection of user media items and / or used in conjunction with a knowledge graph metadata network are described in more detail, for example, in the '269 patent application, which is incorporated by reference above.

[0028] As used herein, "context" and variations thereof refer to any or all attributes of a user device that includes a collection of user media items associated with a user or that can have access thereto, such as physical, logical, social, and other contextual information. As used herein, "contextual information" and variations thereof refer to metadata that describes or defines a user context or a context of a user device that includes a collection of user media items associated with a user or that can have access thereto. Exemplary contextual information includes, but is not limited to, the following: a predetermined time interval; an event scheduled to occur at the predetermined time interval; a geographic location visited during a particular time interval; one or more identified persons associated with a particular time interval; an event that occurred during a particular time interval, or a geographic location visited during a particular time interval; weather metadata that describes weather associated with a particular time period (e.g., rain, snow, sun, temperature, etc.); seasonal metadata that describes a season associated with a capture of one or more user media items; relationship information that describes a nature of a social relationship between a user and one or more third parties; or natural language processing (NLP) information that describes a nature and / or content of an interaction between a user and one or more third parties. For some embodiments, contextual information can be obtained from external sources, such as a social networking application, a weather application, a calendar application, a contacts application, any other type of application, or from any type of data repository that is accessible via a wired or wireless network (e.g., the Internet, a private intranet, etc.).

[0029] Referring again to Figure 1 For one embodiment, the user media item management system 106 uses the user media item metadata 112 to generate a metadata network 114. As Figure 1As shown, all or some of the metadata networks 114 can be stored in the processing unit 104 and / or the memory 110. As used herein, “knowledge graph,” “knowledge graph metadata network,” “metadata network,” and variations thereof refer to a dynamically organized collection of metadata that describes one or more user media items (e.g., one or more groups of user media items in a collection of user media items, one or more user media items in a collection of user media items, etc.) used by one or more computer systems. In a metadata network, no actual user media items are stored, but only metadata (e.g., metadata associated with one or more groups of user media items, metadata associated with one or more user media items, etc.) is stored. Metadata networks differ from databases in that, generally, metadata networks allow for deep connections between metadata using multiple dimensions, which can be traversed for additional inferred relevance. Such deductive reasoning is generally not feasible in conventional relational databases without loading large (e.g., hundreds, thousands, etc.) of database tables. Thus, as noted above, conventional databases can require large amounts of computing resources (e.g., external data stores, remote servers, and their associated communication technology, etc.) to perform deductive reasoning. In contrast, metadata networks can be viewed, manipulated, and / or stored using less computing resource requirements than the conventional databases described above. Further, metadata networks are dynamic resources that have the ability to learn, grow, and adapt to new information added to them. This is different from databases, which are useful for accessing cross-referenced information. While databases can be extended using additional information, they remain a tool for accessing cross-referenced information put into them. Metadata networks do not just access cross-referenced information, but much more, and involve data extrapolation for inferring or determining additional data. As noted above, the user media items themselves can be stored, for example, on one or more servers remote from the client device 101, with thumbnail versions of the user media items stored in the system memory 110, and full versions of particular user media items downloaded and / or stored into the memory 110 of the client device 101 only as needed (e.g., when a user wishes to view or share a particular user media item). However, in other embodiments, the user media items themselves can also be stored within the memory 110, for example, in a separate database such as the conventional databases described above, for example, when the amount of on-board storage space and processing resources at the client device 101 are large enough and / or the size of the user’s collection of user media items is small enough.

[0030] The user media item system 106 can generate a metadata network 114 as a multidimensional network of user media item metadata 112. As used herein, "multidimensional network" and variations thereof refer to a complex graph with multiple relationships. A multidimensional network generally includes multiple nodes and edges. For one embodiment, the nodes represent metadata, and the edges represent relationships or correlations between the metadata. Exemplary multidimensional networks include, but are not limited to, edge-labeled multigraphs, multi-part edge-labeled multigraphs, and multilayer networks.

[0031] In some embodiments, the metadata network 114 includes two types of nodes: (i) moment nodes; and (ii) non-moment nodes. As used herein, a "moment" shall refer to a contextual organization pattern used to group one or more user media items according to an inferred or explicitly defined association between such user media items (e.g., for the purpose of displaying the group of user media items to a user). For example, a moment can refer to eating lunch at a coffee shop in Cupertino, California on March 26, 2018. For example, this moment can be used to identify one or more user media items (e.g., an image, a group of images, a video, a group of videos, a song, a group of songs, etc.) that are associated with visiting the coffee shop on March 26, 2018 (and not with any other event).

[0032] As used herein, a "moment node" refers to a node in a multidimensional network that represents a moment (as described above). As used herein, a "non-moment node" refers to a node in a multidimensional network that does not represent a moment. Thus, a non-moment node can refer to a metadata asset that is associated with one or more user media items that are not moments, e.g., a node that is associated with a particular person, location, multimedia presentation, etc. Additional details regarding possible types of "non-moment" nodes that can be found in an exemplary metadata network can be found, for example, in the '269 application, which is incorporated by reference above.

[0033] For one embodiment, the edges between nodes in the metadata network 114 represent relationships or correlations between the nodes. For one embodiment, the user media item management system 106 updates the metadata network 114 as new metadata 112 is obtained or received by the metadata network 114 and / or as new metadata 112 for user media items in the user's collection of user media items is determined for the user.

[0034] The user media item management system 106 can use the metadata network 114 to manage the user media items 112 associated with user media item metadata in various ways. For a first example, the user media item management system 106 can use the metadata network 114 to identify and present a group of user media items of interest in a collection of user media items based on a correlation between user media item metadata (i.e., nodes in the metadata network 114) and one or more criteria (i.e., edges in the metadata network 114). For this first example, the user media item management system 106 can select media items of interest based on a time node in the metadata network 114. In some embodiments, the user media item management system 106 can recommend one or more media assets (such as audio media) that can be used to accompany a presentation of the selected user media items of interest.

[0035] The client device 101 can also include a memory 110 to store and / or retrieve the metadata 112, the metadata network 114, the candidate media asset metadata 126, and / or optional data 116 described by or associated with the metadata 112. The metadata 112, the metadata network 114, and / or the optional data 116 can be generated, processed, and / or captured by other components in the client device 101. For example, the metadata 112, the metadata network 114, and / or the optional data 116 can include data generated, captured, processed, or associated by one or more peripheral devices 118, the user media item capture device 102, or the processing unit 104, among others. The client device 101 can also include a memory controller (not shown) that includes at least one electronic circuit that manages data flowing into and / or out of the memory 110. The memory controller can be a separate processing unit or integrated in the processing unit 104.

[0036] The client device 101 can include a user media item capture device 102 (e.g., an imaging device to capture images, an audio device to capture sound, a multimedia device to capture audio and video, any other known user media item capture device, etc.). The device 102 is shown in a dashed box to illustrate that it is an optional component of the client device 101. For one embodiment, the user media item capture device 102 can also include a signal processing pipeline implemented as hardware, software, or a combination thereof. The signal processing pipeline can perform one or more operations on data received from one or more components in the device 102. The signal processing pipeline can also provide the processed data to the memory 110, the peripheral devices 118 (as discussed further below), and / or the processing unit 104.

[0037] The client device 101 can also include a peripheral device 118. For one embodiment, the peripheral device 118 can include at least one of: (i) one or more input devices that interact with or send data to one or more components in the client device 101 (e.g., a mouse, a keyboard, etc.); (ii) one or more output devices that provide output from one or more components in the client device 101 (e.g., a monitor, a printer, a display device, etc.); or (iii) one or more storage devices that store data in addition to the memory 110. The peripheral device 118 is shown in phantom to illustrate that it is an optional component of the client device 101. The peripheral device 118 can also refer to a single component or device that can function as both an input device and an output device (e.g., a touch screen, etc.). The client device 101 can include at least one peripheral control circuit (not shown) for the peripheral device 118. The peripheral control circuit can be a controller (e.g., a chip, an expansion card, or a standalone device, etc.) that interacts with the peripheral device 118 and is used to instruct operations performed by the peripheral device. The peripheral device control circuit can be a separate processing unit or integrated in the processing unit 104. The peripheral device 118 can also be referred to throughout this document as an input / output (I / O) device 118.

[0038] The client device 101 can also include one or more sensors 122, which are shown in phantom to illustrate that the sensors can be optional components of the client device 101. For one embodiment, the sensors 122 can detect one or more characteristics of an environment. Examples of sensors include, but are not limited to, a light sensor, an imaging sensor, an accelerometer, a sound sensor, a barometric pressure sensor, a proximity sensor, a vibration sensor, a gyroscope sensor, a compass, a barometer, a thermal sensor, a rotation sensor, a speed sensor, and an inclinometer.

[0039] For one embodiment, client device 101 includes a communication mechanism 120. Communication mechanism 120 can be, for example, a bus, network, or switch. When technology 120 is a bus, technology 120 is a communication system that transfers data between components in client device 101, or between components in client device 101 and other components associated with other systems (not shown). As a bus, technology 120 includes all relevant hardware components (wires, fiber, etc.) and / or software, including communication protocols. For one embodiment, technology 120 can include an internal bus and / or an external bus. Further, technology 120 can include a control bus, address bus, and / or data bus for communication associated with client device 101. For one embodiment, technology 120 can be a network or switch. As a network, technology 120 can be any wired or wireless network, such as a local area network (LAN), wide area network (WAN) such as the Internet, fiber network, storage network, or combination thereof. When technology 120 is a network, components in client device 101 do not have to be physically co-located. Individual components in client device 101 can be directly linked through the network, even though these components can not be physically adjacent to one another. For example, two or more of processing unit 104, communication technology 120, memory 110, peripheral devices 118, sensors 122, and user media item capture device 102 are in different physical locations from one another and are communicatively coupled via communication technology 120, which is a network device or switch that directly links these components through network 140.

[0040] In some cases, client device 101 can be communicatively coupled to server device 150 through network 140. Server device 150 can include electronic components for performing management of media assets and providing media assets to one or more client devices. Server device 150 can be housed in a single computing system, such as a computer server, virtual machine, virtual container, etc., or can be housed in multiple computing systems, such as a computer server system, multiple virtual machines, virtual containers, etc. In some cases, various components of server device 150 can be spatially or logically separated and implemented on separate computing systems that are networked together via an internal network, such as a LAN, WAN, etc. Server device 150 includes one or more network interfaces 152 for communicating with client devices via network 140, and possibly other server devices 150 if so configured.

[0041] In some cases, server device 150 may serve as a media asset repository and access network storage 154. Network storage 154 stores media assets 156 and metadata 158 associated with media assets 156. In some cases, network storage 152 may also include information about events 160 that may occur at certain locations and times. For example, event-related information may include the schedule and location of sporting events, concert locations and times, and the like. Network storage 152 may also include user account information 162, which may include information about users who have access to media assets 156, such as account information, the user's number of visits, which media assets 156 the user has accessed, and the like.

[0042] In some cases, server device 150 may have or may be able to access various services, such as a geohashing service 164 and an audio recognition service 166. Geohashing service 164 may be configured to convert a received geohash into latitude / longitude location information. Audio recognition service 166 may be able to determine what media asset is being played based on an audio clip from the media asset.

[0043] Figure 2 An example of a moment view user interface 230 for presenting a collection of user media items based on the moments during which the user media items were captured is shown, according to one embodiment. Interface 230 includes a list view of user media item collections (in this case, image collections 232, 234, and 236). Each such image collection can represent a unique moment in the user's collection of user media items. Image collections 232, 234, 236 include thumbnail versions of the images presented along with a description of the location at which the image was captured and the date (or date range) at which the image was captured. Temporal data and location data can be used to improve the definition and boundaries between moments to more precisely define moments and to divide moment collections into more specific moments, as described in more detail, for example, in U.S. non-provisional patent application Ser. No. 14 / 733,663, entitled “Using Locations to Define Moments,” filed on June 8, 2015 (“the '663 application”), which is incorporated by reference above.

[0044] Figure 3 An exemplary knowledge graph metadata network 300 according to aspects of the present disclosure is shown in block diagram form. Figure 3 The exemplary metadata network shown can be represented by Figure 1 The user media item management system shown generates and / or uses. For one embodiment, Figure 3 The metadata network 300 shown is combined with the above Figure 1 The metadata network 114 is similar or identical to the metadata network 114. It should be understood that Figure 3 The metadata network 300 depicted and shown is exemplary and does not show every type of node or edge that the user media item management system 106 can generate.

[0045] In Figure 3 In the metadata network 300 shown, nodes representing metadata are shown as circles, and edges representing correlations between metadata are shown as connections or edges between the circles. In addition, certain nodes are labeled with the type of metadata they represent (e.g., region, city, state, country / territory, year, day, week, month, point of interest (POI), area of interest (AOI), region of interest (ROI), person, time of day type, time of day name, performer, activity venue, merchant name, merchant category, etc.). In Figure 3 In the example metadata network 300 shown, the time of day node 302 is shown linking together various other metadata nodes.

[0046] For one embodiment, the metadata represented in the nodes of the metadata network 300 can include, but is not limited to: other metadata such as the user's relationships with other people (e.g., family members, friends, co-workers, etc.), the user's work locations (e.g., past work locations, current work location, etc.), the user's interests (e.g., hobbies, owned user media items, consumed user media items, used user media items, etc.), places visited by the user (e.g., previous places visited by the user, places to be visited by the user, etc.). Such metadata information can be used (or used in combination with other data) to determine or infer at least one of the following: (i) that the user is on vacation or traveling; days of the week (e.g., weekends, holidays, etc.); locations associated with the user; social groups of the user; types of places visited by the user (e.g., restaurants, coffee shops, etc.); categories of events (e.g., cooking, sports, travel, etc.); etc. The foregoing examples are meant to be illustrative and not limiting of the types of metadata information that can be captured in the metadata network 300.

[0047] One or more moments can be organized and presented as a memory, e.g., in the form of a multimedia presentation. In some implementations, a memory can include only a presentation of user media items associated with a single moment. However, in other implementations, a memory can include, for example, user media items associated with two or more related moments. In still other implementations, for example, where a user can have multiple sets of user media items that relate to an asset for reasons other than having similar capture times and capture locations, a memory can alternatively include all user media items within some arbitrary time interval at one or more locations that relate to a particular situation, activity, theme, scene type, person, pet, etc., regardless of how many different moments such user media items can be associated with. As one example, certain photos and videos taken by a user on a beach trip can be selected for inclusion in a memory. As another example, a memory presentation can be based on a person and include user media items in which the person appears. User media items from one or more moments are arranged into a multimedia presentation that can be displayed (e.g., played back) to the user, shared to other users, etc. In some cases, user media items included in a memory can be arranged into a slideshow-type presentation with transitions played between displays of clusters of one or more user media items. User media items presented in a memory can be displayed / played / arranged on a background and captions can be added. In some cases, an accompaniment such as a background, transitions, captions, recommended color processing, etc. can be recommended to the user prior to first presenting or playing back a memory.

[0048] According to aspects of the disclosure, media assets can be recommended to a user to accompany presentation of a memory. For example, one or more music tracks from a media asset repository can be recommended to a user to score playback of a memory. Figure 4 FIG. 4 is a flowchart 400 illustrating techniques for recommending media assets for memory playback according to aspects of the disclosure. It can be appreciated that the concepts embodied in flowchart 400 can be performed by either a client or server device. Performing the steps included in flowchart 400 on a client device helps to protect privacy because less information is transmitted to a server device.

[0049] At block 402, a set of candidate media assets is requested for a set of user media items based on a network of knowledge graph metadata that describes features of the set of user media items. In some cases, a client device can request metadata for candidate media assets based on user media items identified for a memory. The client device can issue a request to one or more media asset repositories. In some cases, the media asset repositories can include multiple media asset repositories and multiple sets of metadata candidate media assets can be requested from multiple media asset repositories.

[0050] In some cases, a user associated with a client device can access a media asset repository using the user's user account, allowing the user to access (e.g., playback) media assets. The media asset repository can be configured to store a list of media assets that the user has accessed. For example, an online media service, such as a video or music service, can store a list of music or videos that the user has previously watched or listened to. In some cases, this list of media assets can be pre-filtered, for example, to remove media assets that the user dislikes (e.g., despises) or that can be associated with media assets that are not associated with the user (e.g., instances of children's songs repeated fifty times in a day). In some cases, the client device can request that the media asset repository recommend media assets for the user for a set of candidate media assets. In some cases, additional sets of candidate media assets can also be requested, as described below. For example, the client device can request candidate media assets based on one or more categories of candidate media assets. These categories can be based on information contained in the metadata network that is associated with the user's media items (e.g., particular scenes, objects, locations, dates, themes, or topics that are identified as being related to or present in the user's media items). In some cases, the client device can request candidate media assets based on multiple categories of candidate media assets and obtain more candidate media assets than can be used for a single memory. For example, candidate media assets can be requested for multiple memories, likely memories, a superset of categories, etc. As a more particular example, the client device can request candidate media assets associated with categories that can be used for likely memories. In some cases, the client device can request candidate media assets associated with all possible categories that are applicable to the user's media items that are accessible to the client device. The client device can store the received candidate media assets for use in memories created later.

[0051] At block 404, metadata for a set of candidate media assets is received. For example, one or more media asset repositories can respond and transmit the requested metadata for the candidate media assets. In some cases, the client device can receive the set of candidate media assets from the one or more media asset repositories and process the candidate media assets into a set of recommended media assets. The client device can receive more candidate media assets than the client device will display to the user.

[0052] At block 406, one or more ranked sets of media assets are determined based on the received metadata. For example, the client can receive metadata for hundreds of candidate media assets and then filter and / or rank the candidate media assets to select a significantly smaller set of media assets (e.g., <10) to recommend to the user. In some cases, different ranking techniques can be used to rank different categories of candidate media assets. At block 408, the determined one or more ranked sets of media assets are output. For example, the selected set of media assets can be displayed to the user as recommended media assets to accompany the presentation of the user's media item in a memory.

[0053] Figure 5 5 is a block diagram 500 illustrating optional aspects for requesting candidate media assets according to aspects of the present disclosure. As shown, at box 402, candidate media assets of different categories may be optionally requested for a user's set of media items. In some cases, the client device may request candidate media assets based on one, more than one, or all available categories. Optionally, at box 502, candidate media assets previously accessed by the user may be requested. In some cases, a media asset repository may store a list of media assets accessed by the user, and metadata related to the media assets previously accessed by the user may be requested. In some cases, the client device may have regularly accessed the list of previously accessed media assets. For example, the list and metadata related to the media assets on the list may be accessed and regularly updated via a media playback application (such as a media player application).

[0054] Optionally, at block 504, metadata related to the candidate media asset may be requested based on time. In some cases, the time may be based on a time period associated with the remembered user media item. As an example, for a user that includes Figure 3 , metadata associated with candidate media assets accessed by the user on or around the date associated with the memory (i.e., March 26, 2018). In some cases, the metadata associated with the candidate media assets may include information related to the manner in which the media assets were accessed. For example, the metadata may include an indication of the number of times a particular media asset was accessed and / or whether the media asset was skipped, deprecated, etc.

[0055] In some cases, the time period associated with the user media items in memory may span a relatively long time period. For example, a memory may be based on a year and include user media items from that year. In such cases, metadata may be requested for candidate media assets within a time range, rather than requesting metadata for candidate media assets that are specific to a time associated with a particular user media item. In some cases, the media asset repository may be configured to periodically determine lists of media assets accessed by a user during a particular time period, and metadata associated with the media assets in these lists may be requested. For example, the media asset repository may generate an annual "review" or "best music for you in [year]" playlist based on the media accessed by the user during the year. When the time period associated with the memory aligns with and / or covers a time period associated with such a generated list, the client device may request the generated list and obtain metadata associated with the media items from the generated list.

[0056] To help maintain privacy, metadata related to candidate media assets based on a time request can be oversampled. For example, to avoid a media asset repository being able to determine a date that a client device created a memory for, a client device request can request candidate media assets based on a time period that is more than a time period associated with the memory. This can help obfuscate the time period associated with the memory. As an example, for a memory including Figure 3 the time shown, in addition to the date associated with the memory (i.e., March 26, 2018), a client device can request metadata related to candidate media assets for multiple days. As another example, in the case of a client device requesting a regularly determined list of media assets that a user accessed within a particular time period, the client device can request determined media assets that the user accessed within multiple time periods (e.g., 2016 review, 2017 review, 2018 review, etc.). In some cases, a client device can request metadata related to candidate media assets within a time period that is consistent with all user media items available to the client device.

[0057] Optionally, at block 506, metadata related to candidate media assets can be requested based on a location. The location can be based on a location associated with a user media item of the memory. In some cases, a client device can indicate a time and a location, and a media asset repository can provide metadata related to candidate media assets based on the indicated time and location. As an example, for a memory including Figure 3 the time shown, a client device can request metadata related to candidate media assets associated with a time and a location (in this example, March 2018, in the San Francisco Bay Area), and a media asset repository can return metadata related to candidate media assets that were popular at the requested time and location. In some cases, a server can first remove media assets that were popular in a larger area (e.g., international or global popular) media assets from the returned candidate media assets at the requested time and location. This helps preserve a local flavor for the returned candidate media assets at the requested time and location. As an example, for a memory of a foreign travel, the candidate media assets returned at the requested time and location can include media assets that were popular while on the foreign travel. Media assets that were popular internationally at the time (or popular in a country / region associated with the user) can be removed to help link the memory to the foreign country.

[0058] To help maintain privacy, the time and location data can be oversampled. For example, to avoid the media asset repository being able to determine the time / location of the client device, the client device can request candidate media assets based on a time period in which the client device was not in a particular location. For example, the client device can request candidate media assets for a foreign country for a random number of days before and after a time period associated with a remembered user media item. In some cases, the location information sent to the media asset repository can be geohashed with reduced precision. For example, latitude / longitude coordinate location information can be geohashed with one digit of precision to identify an area of approximately 2,500 square kilometers. This reduced precision allows identification of a general area in which the client device is located, such as the San Francisco Bay Area of California, but is not precise enough to pinpoint the client device. The metadata for the returned candidate media assets can be based on this reduced precision location information. For example, the media asset repository can return metadata for all media assets that are popular in the general area identified by the reduced precision location information.

[0059] Optionally, at block 508, metadata related candidate media assets can be requested based on one or more events. The event information can be based on a remembered user media item. In some cases, the client device can indicate a time and location, and the media asset repository can determine one or more events that occurred at the provided time and location. The media asset repository can then provide metadata related to candidate media assets based on the one or more events that occurred at the provided time and location. For example, a memory, such as attending a concert, can include a user media item associated with a date and location. The client device can request metadata related to candidate media assets associated with the time and location information, and the media asset repository can provide information related to events that occurred at that time and location. For example, metadata for candidate media assets related to the concert, such as music by artists that performed at the concert, can be provided. In some cases, to help maintain privacy, as described above, the time can include one or more times, or can be a range of times, and the location information can be geohashed. The media asset repository can then provide information related to all events that occurred across areas at the one or more times or during the range of times.

[0060] In some cases, one or more media assets can be identified in a user media item, and metadata related to media assets associated with the identified one or more media assets can be provided. As an example, a memory can include a user media item, such as a video that includes a portion of a song in the background. The portion of the song can be analyzed by one or more online services to identify the song. In some cases, the online services can be provided by the media asset repository. Metadata related to the identified song and associated with media assets that can include the identified song can be provided to the client device.

[0061] Optionally, at block 510, metadata-related candidate media assets can be requested based on one or more themes. The themes can be based on themes associated with the user's media items of the memory. In some cases, one or more themes can be inferred from the user's media items selected for presentation as the memory. In some cases, the themes can be inferred based on a metadata network associated with the user's media items. For example, as discussed above, a user's relationships with others, their interests, places visited, cities, regions, etc. can be represented in a metadata network, and based on these representations, holidays, trips, locations, types of places visited, categories of events, etc. can be inferred. This represented and inferred information can be used alone or in combination to determine one or more themes of the selected user's media items. Returning to the example memory shown at the moment in time, the client device can determine that the theme of the memory is lunch with friends based on the metadata network. Then, the client device can request metadata of candidate media assets that can be used to accompany the memory with the theme of lunch with friends. In some cases, a set of themes can be predefined. In some cases, multiple themes can be identified for a set of user's media items. For example, the client device can determine that the user's media items of the memory can include a theme of a beach visit, and since the visit was during a holiday, such as Thanksgiving, another theme of the memory can be the Thanksgiving holiday. In some cases, candidate media assets can be requested based on one or more determined themes. For example, the client device can send a request to a media asset repository including an indication of the determined themes. Figure 3

[0062] ​In some cases, the media asset repository can include one or more media assets that are tagged with metadata corresponding to the determined one or more themes. For example, certain songs on the media asset repository can be identified and tagged as suitable for inclusion in a memory in the theme of a beach or a Thanksgiving holiday user media project. In some cases, media assets can be identified and tagged as suitable for certain themes by a human curator. In other cases, media assets can be identified as suitable for certain themes based on machine learning techniques and / or a combination of human curators and machine learning. In some cases, how a media asset's metadata tags match the determined one or more themes can be based in part on how well the media asset matches the determined one or more themes. How well the metadata tags match the determined one or more themes can be based on any suitable known technique. For example, determining how well the metadata tags match the themes can include counting a number of matching tags and themes, or can include one or more machine learning models trained to correlate tags that can not match perfectly. In some cases, certain tags can be considered more important than other tags. For example, where the themes include certain holidays, media assets with a corresponding holiday tag can be considered to match better even if other tags of those media assets do not match the determined themes.

[0063] In some cases, media assets can be identified and tagged based in part on how well the media assets fit the theme of the companion user media project based on one or more factors. As an example, for music media assets, these factors can include the length of the song, the length of the intro, characteristics of the song (e.g., energy, mood, etc.), and / or lyrics analysis. In some cases, portions of a media asset can be identified as fitting better than other portions of the media asset. For example, a chorus portion of a song can be identified as fitting very well, while an intro portion of the song can not fit well.

[0064] To help maintain privacy, themes can be oversampled. For example, to avoid the media asset repository being able to determine the theme of a particular memory being created by a client device, the client device can request candidate media assets associated with themes other than the theme associated with the memory being created.

[0065] As described above, the client device can request more metadata for candidate media assets than are ultimately presented to the user for selection. Figure 6is a block diagram 600 illustrating optional aspects for determining one or more ranked media asset sets, in accordance with aspects of the present disclosure. After receiving metadata for a set of candidate media assets, at block 406, a determination of one or more ranked media asset sets is made based on the received metadata. While the determination of one or more ranked media asset sets can be made by an online service, such as a media asset repository, to help maintain privacy, the determination of one or more ranked media asset sets can be performed by a client device.

[0066] Optionally, at block 602, the set of candidate media assets can be filtered. In some cases, the set of candidate media assets can be filtered to remove some media assets. For example, the set of candidate media assets can be filtered to remove media assets containing explicit content based on the length of a particular media asset, digital rights management issues regarding a particular media asset, etc. In some cases, the filter can be user configurable. As discussed above, the requested candidate media assets can be oversampled. In some cases, filtering can be performed to remove oversampled candidate media assets returned from the media asset repository.

[0067] Optionally, at block 604, the set of candidate media assets can be ranked. Ranking can be used to help identify the most potentially relevant media assets in the set of candidate media assets in order to present them to the user. Ranking can be performed based on one or more ranking factors. Optionally, at block 606, the candidate media assets can be ranked based on compatibility values. In some cases, the metadata for the media assets in the set of candidate media assets can include a compatibility value for each media asset indicating the compatibility of the media asset with the user. As an example, a media asset repository can recommend one or more media assets to a user based on media assets consumed by the user. As a more particular example, a music or video service can recommend other music or videos to a user based on content previously listened to or watched by the user. The other music or videos can include content previously consumed by the user, or content not previously consumed by the user. In some cases, to recommend media assets to a user, a media asset repository can determine compatibility values for the user for media assets of the media asset repository, the compatibility values indicating, for example, the likelihood that the media asset will be appealing to the user. These compatibility values can be included in the metadata for the set of candidate media assets.

[0068] Optionally, at block 608, the candidate media assets can be ranked based on the energy of the candidate media assets. In some cases, the metadata of the media assets in the set of candidate media assets can include an indication of the energy level of the media asset. The energy level of a media asset can be a measure of the pace at which the media asset is perceived based on the characteristics of the media asset. For example, the energy level of a song can be determined based on a measure of the melody of the song, the sound of the song, the number of beats per minute, etc. Continuing with this example, a song with a more beautiful melody, a higher sound score, and a lower number of beats per minute can be determined to have a lower energy.

[0069] In some cases, the energy level can be inferred from the user media items selected for presentation as a memory. The energy level can be inferred based on the metadata network associated with the user media items. As discussed above, the metadata network can include representation and inferred information about the user media items, and the representation and inferred information can be used individually or in combination to determine the energy level of the selected user media items. In some cases, the determined energy level can be based on one or more themes or lyrical content of the selected user media items. Returning to the example memory shown in FIG. 6B, the client device can determine, based on the metadata network, that the selected user media items can be associated with a lower energy level when the selected user media items are related to lunch with a friend. Similarly, the set of selected user media items related to a snowboarding trip can be associated with a higher energy level. Figure 3

[0070] Once the energy level associated with the selected user media items is determined, the media assets of the candidate media assets can be ranked based on how the energy level of the media assets of the candidate media assets matches the energy level associated with the selected user media items. How the energy level matches can be determined based on any known technique. In some cases, it can not be appropriate for a user to match higher energy media assets with selected user media items associated with a lower energy. Conversely, in many cases, it can work well to match lower energy media assets with selected user media items associated with a higher energy. Thus, media assets with a relatively low energy level can be preferred compared to selected user media items.

[0071] ​Optionally, at block 610, the candidate media assets can be ranked based on their relevance to the selected user media item. In some cases, the relevance of a media asset can be based on one or more cultural factors. In some cases, cultural factors can differ for different countries and regions. For example, a cultural factor for a user can be defined based on where the user typically resides. For example, if a user resides in Quebec, Canada, French language media assets can be ranked higher than media assets in other languages. Additional factors can be applied based on, for example, a determination that the user is visiting another country. For example, if the user media item selected for presentation as a memory is provided for a user from Quebec and is associated with a location in the Bahamas, the ranking of candidate media assets suitable for accompanying the user media item that originates from a beach theme in Canada can be lower than the ranking of candidate media assets suitable for accompanying the user media item that originates from a beach theme in the Bahamas.

[0072] In some cases, the ranking of certain media assets can be adjusted, for example, based on whether the media asset has been previously output and / or selected by the user. For example, a media asset can be highly ranked by any one or more of the ranking techniques, and previously output as a recommended media asset to the user. If the media asset has been previously output multiple times without being selected by the user, the ranking of the media asset can be adjusted to lower the ranking of the media asset. As another example, if another media asset has been previously recommended to the user and selected by the user for another memory, the ranking of the media asset can be adjusted to lower the ranking of the media asset to help encourage the user to select a different media asset.

[0073] In some cases, the manner in which candidate media assets are ranked can differ depending on the category of candidate media assets requested. For example, candidate media assets requested based on one or more themes associated with the user media item for a memory can be ranked based on compatibility values. In some cases, candidate media assets can also be ranked based on multiple ranking techniques. For example, candidate media assets requested based on the user’s previous visits can first be ranked based on the energy of the candidate media assets. Then, the candidate media assets can be ranked again based on compatibility values, for example, for media assets having the same energy metric or within a range of energy metrics.

[0074] A set of recommended media assets can be determined based on the ranked media assets output. The set of recommended media assets can be displayed to the user for selection to accompany the user media item for a memory. To help increase the diversity of recommended media assets output to the user for selection, multiple sets of recommended media assets can be output. In some cases, the multiple sets of recommended media assets can be based on one or more categories of candidate media assets requested.

[0075] Figure 7An exemplary user interface 700 for displaying recommended media assets is shown in accordance with aspects of the present disclosure. Any number of sets of recommended media assets can be included in the user interface 700. As shown, the user interface 700 includes three sets of recommended media assets. If additional sets of recommended media assets are available in the user interface 700, those additional sets can be accessed, e.g., by scrolling, swiping, etc.

[0076] In some cases, the requested multiple categories of candidate media assets can be combined into a single set of recommended media assets. In this example, the first set of recommended media assets (here, Best picks 702) can include high ranking candidate media assets requested based on one or more topics associated with the user's remembered media items and high ranking candidate media assets previously accessed by the user. The first number of highest ranking candidate media assets based on the topic request and the second number of highest ranking candidate media assets previously accessed by the user can be included in Best picks 702. In some cases, one category can be preferred over another when the multiple categories of candidate media assets are combined. For example, the first number of candidate media assets based on the topic request can be displayed before the candidate media assets previously accessed by the user. In other cases, the categories can be interleaved, or the candidate media assets in the set of recommended media assets can be ranked. In some cases, the manner in which the categories are combined can be based on the user's media items. For example, if the topic associated with the user's remembered media items is a particular holiday, candidate media assets of one category, such as candidate media assets based on the topic (e.g., the particular holiday) request, can be prioritized in the set of recommended media assets. In some cases, the user can access a complete list of candidate media assets in the set of recommended media assets by accessing a UI element (here, View all 704).

[0077] In this example, the second set of recommended media assets (Recommended for you 706) can include candidate media assets of a single category. Here, Recommended for you 706 includes high ranking candidate media assets previously accessed by the user. In cases where media assets in one category of candidate media assets are included in another set of recommended media assets, those media assets can not be included in the other set of recommended media assets. For example, the Best picks 702 set of selected media assets includes the second number of candidate media assets previously accessed by the user. The candidate media assets included in the Best picks 702 set of selected media assets can not be included in the Recommended for you 706 set of selected media assets. Instead, the next highest ranking candidate media assets can be included in the Recommended for you 706 set of selected media assets.

[0078] As another example, a third set of recommended media assets (from this point forward 708) can include high ranking candidate media assets based on a temporal request. Other sets of recommended media assets can be based on, for example, candidate media assets based on an event request or candidate media assets based on a location request. In some cases, the sets of recommended media assets can be user configurable.

[0079] In some cases, the sets of recommended media assets can vary based on aspects of the remembered user media item. For example, where the time period associated with the remembered user media item spans a long time period, candidate media assets based on a temporal request can be less useful, as there can be too many media assets from that time period. In such cases, the set of recommended media assets based on a time period can be omitted.

[0080] In some cases, once a user selects a media asset for accompanying a remembered user media item, the media asset can be retrieved from the media asset store for playback. In other cases, an identifier associated with the selected media asset can be associated with the memory, such that the selected media asset can be retrieved from the media asset store by its identifier at a later point in time (e.g., when the user next requests playback of the memory). In some cases, for example, as a user's musical tastes change, the user can also be presented with a user interface option to change or update the selected media asset for a given memory over time, and / or the user (or media management system) can be able to identify and discover another audio media asset that the user can find more suitable to use as a soundtrack for a given memory.

[0081] Referring now to Figure 8 , a simplified functional block diagram of an illustrative programmable electronic device 800 for performing user media item management is shown, in accordance with one embodiment. The electronic device 800 can be, for example, a system of a mobile phone, a personal media device, a portable camera, or a tablet, notebook, or desktop computer. As shown, the electronic device 800 can include a processor 805, a display 810, a user interface 815, graphics hardware 820, device sensors 825 (e.g., a proximity sensor / ambient light sensor, an accelerometer, and / or a gyroscope), a microphone 830, an audio codec 835, a speaker 840, communication circuitry 845, image capture circuitry or units 850 (e.g., which can include multiple camera units / optical sensors of different characteristics (as well as camera units housed outside of the device 800 but in electronic communication with the device)), a video codec 855, a memory 860, a storage 865, and a communication bus 870.

[0082] The processor 805 can execute instructions necessary for performing or controlling operations of various functions performed by the device 800 (e.g., such as generation and / or processing of user media items in accordance with various embodiments described herein). The processor 805 may, for example, drive the display 810 and can receive user input from the user interface 815. The user interface 815 can take various forms, such as buttons, a keypad, a dial, a click wheel, a keyboard, a display screen, and / or a touch screen. The user interface 815 may, for example, be a wire through which a user can view a captured video stream and / or indicate a particular image that the user wants to capture or share (e.g., by clicking a physical or virtual button at the moment the desired image is being displayed on the device's display screen).

[0083] In one embodiment, the display 810 can display a video stream as it is being captured while the processor 805 and / or the graphics hardware 820 and / or the image capture circuitry is storing the video stream (or individual image frames from the video stream) in the memory 860 and / or the storage 865. The processor 805 can be a system on a chip such as those found in mobile devices, and can include one or more specialized graphics processing units (GPUs). The processor 805 can be based on a reduced instruction set computer (RISC) or complex instruction set computer (CISC) architecture, or any other suitable architecture, and can include one or more processing cores. The graphics hardware 820 can be specialized computing hardware for processing graphics and / or assisting the processor 805 in performing computational tasks. In one embodiment, the graphics hardware 820 can include one or more programmable graphics processing units (GPUs).

[0084] For example, in accordance with the present disclosure, image capture circuitry 850 can include one or more camera units configured to capture images, e.g., images that can be managed by the user media item system. Output from image capture circuitry 850 can be at least partially processed by video codec 855 and / or processor 805 and / or graphics hardware 820 and / or a dedicated image processing unit incorporated within circuitry 850. Images so captured can be stored in memory 860 and / or storage 865. Memory 860 can include one or more different types of media used by processor 805, graphics hardware 820, and image capture circuitry 850 to perform device functions. For example, memory 860 can include memory cache, read-only memory (ROM), and / or random access memory (RAM). Storage 865 can store media (e.g., audio files, image files, and video files), computer program instructions or software, preference information, device profile information, and any other suitable data. Storage 865 can include one or more non-transitory storage media including, for example, magnetic disks (fixed, floppy, and removable) and magnetic tapes, optical media such as CD-ROMs and digital video disks (DVDs), and semiconductor memory devices such as Electrically Programmable Read-Only Memories (EPROM), and Electrically Erasable Programmable Read-Only Memories (EEPROM). Memory 860 and storage 865 can be used to hold computer program instructions or code organized into one or more modules and written in any desired computer programming language. Such computer program code, when executed by, for example, processor 805, can implement one or more of the methods described herein. Power source 875 can include a rechargeable battery (e.g., lithium-ion battery, etc.) or other electrical connection to a power source (e.g., to an electrical outlet) for managing and / or providing power to the electronic components and associated circuitry of electronic device 800.

[0085] In the foregoing description, many specific details such as specific configurations, properties and processes are described to provide a thorough understanding of the embodiments. In other examples, well-known processes and manufacturing techniques have not yet been described in particular detail so as not to unnecessarily obscure the embodiments. Throughout this specification, references to "one embodiment," "another embodiment," "other embodiments," "some embodiments," and variations thereof mean that the specific features, structures, configurations, or characteristics described in conjunction with the embodiment are included in at least one embodiment. Therefore, the phrases "for one embodiment," "for an embodiment," "for another embodiment," "in other embodiments," "in some embodiments," or variations thereof appearing throughout this specification do not necessarily refer to the same embodiment. In addition, specific features, structures, constructions, or characteristics may be combined in one or more embodiments in any appropriate manner.

[0086] In the following description and claims, the terms "coupled" and "connected" and their derivatives may be used. It should be understood that these terms are not intended to be synonymous with each other. "Coupled" is used herein to indicate that two or more elements or components that may or may not be in direct physical or electrical contact with each other cooperate or interact with each other. "Connected" is used to indicate the establishment of communication between two or more elements or components that are coupled to each other.

[0087] Certain portions of the foregoing detailed description have been presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. An algorithm, as used herein and generally, refers to a self-consistent sequence of operations leading to a desired result. These operations are those requiring physical manipulation of physical quantities. However, it should be kept in mind that all of these and similar terms are associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless otherwise specifically stated, it will be apparent from the foregoing discussion that discussions throughout this specification utilizing terms such as those set forth in the following claims will refer to operations and processes of a computer system or similar electronic computing system that can manipulate data represented as physical (electronic) quantities within the computer system's registers and memories and convert them into other data that are also displayed as physical quantities within the computer's memories, registers, or other such information storage, transmission, or display devices.

[0088] Embodiments described herein can relate to an apparatus for performing a computer program (e.g., operations described herein, etc.). Such computer programs can be stored in a non-transitory computer-readable medium. A machine-readable medium includes any mechanism for storing information in a form readable by a machine (e.g., a computer). For example, a machine-readable (e.g., computer-readable) medium includes a machine (e.g., a computer) readable storage medium (e.g., read only memory (“ROM”), random access memory (“RAM”), magnetic disk storage media, optical storage media, flash memory devices).

[0089] Although operations or methods are described above according to some sequential order, it should be understood that some of the operations described can be performed in a different order. Moreover, some operations can be performed in parallel rather than sequentially. The embodiments described herein are not described with reference to any particular programming language. It will be appreciated that a variety of programming languages can be used to implement the various embodiments of the disclosed subject matter. In utilizing various aspects of the embodiments described herein, those skilled in the art will appreciate that a combination of the above-described embodiments, or variations or modifications thereof, can be used for managing components of a processing system to increase power and performance of at least one of those components. Accordingly, it is clear that various modifications can be made to the disclosed concepts without parting from the broader spirit and scope of at least one of the disclosed subject matter set forth in the appended claims. The specification and drawings are, accordingly, to be regarded as illustrative rather than restrictive.

[0090] In the development of any actual implementation (e.g., such as software and / or hardware development projects, etc.) of one or more of the disclosed concepts described herein, numerous decisions must be made regarding implementation-specific aspects of the development, e.g., specific hardware and / or software modules, components, and the like. These implementation-specific aspects, and the specific choices made for them, can vary depending on the specific implementation and / or objectives of the development. Such development work, while perhaps being complex and time-consuming, can nevertheless be a routine undertaking for those of ordinary skill in the art having the benefit of this disclosure and having access to the teachings presented herein.

[0091] As described above, one aspect of the present technology is the gathering and use of data from a variety of sources to improve the delivery of content sharing suggestions to users. The present disclosure contemplates that, in some instances, this gathered data can include personal information data that uniquely identifies or can be used to contact or locate a specific person. Such personal information data can include demographic data, location-based data, telephone numbers, email addresses, home addresses, social network account identifiers, health or health-related data, birth date, or any other identifying or personal information.

[0092] The present disclosure recognizes that the use of such personal information data, in the present technology, can be used to the benefit of users. For example, the personal information data can be used to deliver targeted content sharing suggestions that are of greater interest and / or greater context relevance to the user. Accordingly, use of such personal information data enables users to have a more streamlined and meaningful experience with content sharing with others. Further, other uses for personal information data that benefit the user are also contemplated by the present disclosure. For example, health and fitness data can be used to provide insights into a user's overall health status, or health status during various times or events in their life.

[0093] The present disclosure contemplates that the entities responsible for the collection, analysis, disclosure, transfer, storage, or other use of such personal information data will comply with well-established privacy policies and / or privacy practices. In particular, such entities should implement and consistently use privacy policies and practices that are generally recognized as meeting or exceeding industry or governmental requirements for maintaining the privacy of personal information data. Such policies should be easily accessible and should be updated as the collection and / or use of data changes. Personal information from users should be collected for legitimate and reasonable uses of the entity and not shared or sold outside of those legitimate uses. Further, such collection / sharing should occur after receiving the informed consent of the users. Additionally, such entities should consider taking any needed steps for safeguarding and securing access to such personal information data and ensuring that others with access to the personal information data adhere to their privacy policies and procedures. Further, such entities can subject themselves to evaluation by third parties to certify their adherence to widely accepted privacy policies and practices. In addition, policies and practices should be adapted to the particular types of personal information data being collected and / or accessed and adapted to the specific relationship the users have with the entity collecting and / or accessing that data. For instance, health information data should be handled differently from online website interactions using cookies. Policies should be adapted to the specific types of personal information data collected and / or accessed, and to the specific relationship the users have with the entity responsible for collecting and / or accessing that data.

[0094] Regardless of the foregoing, the present disclosure also contemplates embodiments in which users selectively block the use of, or access to, personal information data. That is, the present disclosure contemplates that hardware and / or software elements can be provided to prevent or block access to such personal information data. For example, in the case of content-sharing recommendation services, the present technology can be configured to allow users to opt-in or opt-out of participation in the collection of personal information data during registration for or through the services at any time. As another example, users can be provided with an option to not provide their content and other personal information data to improve the content-sharing recommendation services. As another example, users can be provided with an option to limit the length of time that a third party maintains their personal information data, the length of time that content-sharing recommendations can be extracted from, and / or completely prohibit the development of a knowledge graph or other metadata profile. In addition to providing "opt in" and "opt out" options, the present disclosure contemplates providing notifications relating to the access or use of personal information. For instance, a user can be notified upon download of an application that their personal information data will be accessed. Then, the user can be prompted for confirmation prior to the application accessing the personal information data.

[0095] Moreover, it is the intent of the present disclosure that personal information data should be managed and handled in a manner that minimizes risks of unintentional or unauthorized access or use. Risk can be minimized by limiting the collection of data and deleting data once it is no longer needed. In addition, and when applicable, including in certain health related applications, data de-identification can be used to protect a user's privacy. De-identification can be facilitated, when appropriate, by removing specific identifiers (e.g., date of birth, etc.), controlling the amount or

[0096] Thus, while the present disclosure has been broadly categorized as using personal information data to enable one or more various disclosed implementations, the present disclosure also contemplates that the various implementations can also be implemented without the need for access to such personal information data. That is, various implementations of the present technology can not function properly without the presence of all or some of such personal information data, but equally can function properly in the absence of such personal information data. For example, content can be recommended for sharing with a user by inferring preferences from non-personal information data or a bare minimum of personal information such as a quality level of content (e.g., focus, exposure level, etc.) or the fact that a device associated with a user's contact, other non-personal information available to the user media item management system, or publicly available information is requesting certain content.

[0097] As used in the foregoing and in the claims, the phrases "at least one of A, B, or C" and "one or more of A, B, or C" include A alone, B alone, C alone, combinations of A and B, combinations of B and C, combinations of A and C, and combinations of A, B, and C. That is, the phrases "at least one of A, B, or C" and "one or more of A, B, or C" mean A, B, C, or any combination thereof, such that one or more of a group of elements consisting of A, B, and C should not be construed as requiring at least one of each of the listed elements of A, B, and C, regardless of whether A, B, and C are related or otherwise associated. Furthermore, the use of the articles "a" or "the" in introducing an element should not be construed as excluding the presence of more than one of the element. Also, the recitation "A, B, and / or C" means at least one of A, B, or C. Also, "one" as used in the present disclosure means "one or more." For example, "a user media item" means "one user media item" or "a group of user media items."

Claims

1. An electronic device comprising: Memory; and one or more processors operatively coupled to the memory, wherein the one or more processors are configured to execute instructions that cause the one or more processors to: requesting a candidate set of media assets for a user set of media items based on a knowledge graph metadata network describing the user set of media items, wherein the candidate set of media assets includes one or more songs to be used as a soundtrack for the user set of media items; determining a first energy metric for the set of user media items, wherein the first energy metric is based on one or more themes of the set of user media items; receiving metadata for the set of candidate media assets, wherein the metadata includes: (1) an indication of a compatibility value for each candidate media asset in the set of candidate media assets, the compatibility value estimating whether the candidate media asset is compatible with a user; and (2) a second energy metric for each candidate media asset in the set of candidate media assets, wherein the second energy metric is based on one of: a melody of the candidate media asset, acoustics of the candidate media asset, or beats per minute; Ranking the set of candidate media assets based at least in part on: (a) a compatibility value for each of the candidate media assets in the set of candidate media assets; (b) a second energy metric for each of the candidate media assets in the set of candidate media assets; and (c) the first energy metric for the set of user media items; determining one or more ranked media asset sets based on the received metadata and the rankings of the candidate media asset sets; and The determined one or more ranked media asset sets are output.

2. The apparatus of claim 1 , wherein the candidate media assets are requested based on at least one of: media assets previously accessed by the user; a time period associated with the set of user media items; a geographic region associated with the user's set of media items; and A topic associated with the set of user media items determined based on the knowledge graph metadata.

3. The apparatus of claim 2, wherein the candidate media assets are requested based on the time period and the geographic region, and wherein the received metadata for the set of candidate media assets is based on one or more events associated with the time period and the geographic region.

4. A device according to claim 3, wherein an event in the one or more events is a concert, wherein the received metadata includes an indication of an artist performing at the concert, and wherein the ranked media asset set in the one or more ranked media asset sets includes one or more media assets associated with the artist.

5. The apparatus of claim 1, wherein determining the first energy metric for the set of user media items is based on the knowledge graph metadata network. 6 . The apparatus of claim 1 , wherein ranking the set of candidate media assets comprises prioritizing candidate media assets in the set of candidate media assets that are associated with an energy metric that is lower than the first energy metric of the set of user media items. 7 . The device of claim 1 , wherein the one or more processors are further configured to execute instructions that cause the one or more processors to filter the candidate media assets based on the received metadata.

8. A method for recommending media assets, comprising: requesting a candidate set of media assets for a user set of media items based on a knowledge graph metadata network describing the user set of media items, wherein the candidate set of media assets includes one or more songs to be used as a soundtrack for the user set of media items; determining a first energy metric for the set of user media items, wherein the first energy metric is based on one or more themes of the set of user media items; receiving metadata for the set of candidate media assets, wherein the metadata includes: (1) an indication of a compatibility value for each candidate media asset in the set of candidate media assets, the compatibility value estimating whether the candidate media asset is compatible with a user; and (2) a second energy metric for each candidate media asset in the set of candidate media assets, wherein the second energy metric is based on one of: a melody of the candidate media asset, acoustics of the candidate media asset, or beats per minute; Ranking the set of candidate media assets based at least in part on: (a) a compatibility value for each of the candidate media assets in the set of candidate media assets; (b) a second energy metric for each of the candidate media assets in the set of candidate media assets; and (c) the first energy metric for the set of user media items; determining one or more ranked media asset sets based on the received metadata and the rankings of the candidate media asset sets; and The determined one or more ranked media asset sets are output.

9. The method of claim 8, wherein requesting candidate media assets is based on at least one of: media assets previously accessed by the user; a time period associated with the set of user media items; a geographic region associated with the user's set of media items; and A topic associated with the set of user media items determined based on the knowledge graph metadata.

10. The method of claim 9, wherein the candidate media assets are requested based on the time period and the geographic region, and wherein the received metadata for the set of candidate media assets is based on one or more events associated with the time period and the geographic region.

11. A method according to claim 10, wherein an event in the one or more events is a concert, wherein the received metadata includes an indication of an artist performing at the concert, and wherein the ranked media asset set in the one or more ranked media asset sets includes one or more media assets associated with the artist.

12. The method of claim 8, wherein determining the first energy metric for the set of user media items is based on the knowledge graph metadata network.

13. The method of claim 8, wherein ranking the set of candidate media assets comprises prioritizing candidate media assets in the set of candidate media assets that are associated with an energy metric that is lower than the first energy metric of the set of user media items.

14. The method of claim 8, further comprising filtering the candidate media assets based on the received metadata.

15. A non-transitory computer-readable medium storing instructions that, when executed, cause one or more processors of a device to: requesting a candidate set of media assets for a user set of media items based on a knowledge graph metadata network describing the user set of media items, wherein the candidate set of media assets includes one or more songs to be used as a soundtrack for the user set of media items; determining a first energy metric for the set of user media items, wherein the first energy metric is based on one or more themes of the set of user media items; receiving metadata for the set of candidate media assets, wherein the metadata includes: (1) an indication of a compatibility value for each candidate media asset in the set of candidate media assets, the compatibility value estimating whether the candidate media asset is compatible with a user; and (2) a second energy metric for each candidate media asset in the set of candidate media assets, wherein the second energy metric is based on one of: a melody of the candidate media asset, acoustics of the candidate media asset, or beats per minute; Ranking the set of candidate media assets based at least in part on: (a) a compatibility value for each of the candidate media assets in the set of candidate media assets; (b) a second energy metric for each of the candidate media assets in the set of candidate media assets; and (c) the first energy metric for the set of user media items; determining one or more ranked media asset sets based on the received metadata and the rankings of the candidate media asset sets; and The determined one or more ranked media asset sets are output.

16. The non-transitory computer-readable medium of claim 15, wherein the candidate media assets are requested based on at least one of: media assets previously accessed by the user; a time period associated with the set of user media items; a geographic region associated with the user's set of media items; and A topic associated with the set of user media items determined based on the knowledge graph metadata.

Images