Behavioral curation of media assets

Abstract

The present disclosure relates to behavioral curation of media assets. In some implementations, a computing device can create a semantic map that includes identified features that occur in a particular percentage of assets in a subset of assets of a media library. Additionally, the computing device can use the semantic map to analyze assets of the media library to generate semantic scores that can be used to determine a first tier of assets from the media library that have the highest semantic score rating of all assets. When viewing assets of the media library, the computing device can prominently present at least one of the first tier of assets in a user interface.

Description

[0001] This application is a divisional application of Chinese patent application 202010361099.3, filed on April 30, 2020, entitled “Behavioral Curation of Media Assets”. Technical Field

[0002] This disclosure relates in general to the display of media library assets, and more specifically to filtering and selecting specific media assets to be displayed based on user behavior. Background Technology

[0003] A typical media library displays all the images and videos it contains across one or more interfaces. Images and videos can be sorted according to parameters such as capture time, alphabetical order of filenames, etc. However, for large media libraries containing hundreds or thousands of images and videos, this presentation can be too cumbersome to manage, too slow to load, and too tedious for users to navigate to find the images and videos they need.

[0004] Furthermore, users may be very interested in many images and videos, but due to the cropped file size of images and videos stored in the media library, users may not normally encounter these interesting images and videos to view. Therefore, these interesting images and videos may remain hidden in the media library and not be fully utilized. Summary of the Invention

[0005] In some implementations, the computing device may create a semantic map that includes identified features appearing in a specific percentage of assets within a subset of assets in the media library. Additionally, the computing device may use the semantic map to analyze the assets of the media library to generate a semantic score, which can be used to determine the highest-rated Tier 1 asset from the media library across all assets. When viewing the assets of the media library, the computing device may prominently display at least one Tier 1 asset from the Tier 1 assets in the user interface.

[0006] This specific implementation offers at least the following advantages. By selecting user-specific assets to display based on past user interactions with other assets in the media library, it's possible to show user assets that might have been hidden or were previously undisplayed but are predicted to be favored by the user. This eliminates the need for users to sort through numerous assets to find one that is pleasing to the eye or has special meaning to the user among all assets in the media library. This saves users time and effort and avoids frustration when trying to access meaningful assets from the media library.

[0007] Details of one or more specific embodiments are set forth in the following figures and detailed descriptions. Other features, aspects, and potential advantages will become apparent from the detailed descriptions, the figures, and the claims. Attached Figure Description

[0008] Figure 1 A block diagram of an exemplary system for curating media library assets.

[0009] Figure 2 An exemplary system for tracking user interactions with assets in a media library is shown.

[0010] Figure 3 An exemplary system is shown for creating semantic maps to predict which specific assets a particular user will like to view in a media library.

[0011] Figure 4 An exemplary day view of the media library application is shown.

[0012] Figure 5 An exemplary moon view for a media library application is shown.

[0013] Figure 6 An exemplary year view for a media library application is shown.

[0014] Figure 7 A flowchart illustrating an exemplary method for determining the significance of assets in a media library.

[0015] Figure 8 A flowchart of an exemplary method for determining a subset of assets in a media library for semantic mapping.

[0016] Figure 9 A flowchart illustrating an exemplary method for determining the best asset to display based on a theme.

[0017] Figure 10 To be achievable Figures 1 to 9 A block diagram of an exemplary computing device illustrating its features and processes.

[0018] Similar reference symbols in the various figures indicate similar elements. Detailed Implementation

[0019] With the advent of digital cameras, large media libraries of photos, images, videos, and other media assets have become commonplace. More specifically, these large media libraries have become widespread due to the combination of digital cameras with mobile devices (such as smartphones and other electronic devices) that are easy for users to transport and carry daily, allowing users to capture photos and videos to document their lives.

[0020] As used in this article, media assets can include digital photographs and images, videos, animated images, composite presentations, and compilations. Large media libraries can include hundreds or even thousands of individual images and videos collected and stored in one or more locations by one or more users over many years. When a user attempts to access a large media library, many problems may arise that are associated with the large number of images and videos included, such as difficulty in locating the desired image or video among the numerous assets stored in the library, slow loading times for individual assets, slow response times for user interactions with the media library (such as scrolling through images and videos, selecting specific images to display, switching between display types, etc.), and loss of tracking after a photo is captured (forgetting that a photo was taken after it has been taken, etc.).

[0021] Furthermore, users typically do not want to view all of their images, videos, and other assets from their media library or through other applications. Instead, users usually want to view assets that are specific to a single user and have particular significance and / or aesthetic appeal. These best images and videos for one user may differ from those determined to be best for another user. Therefore, in the examples described herein, each user's media library can be managed individually for a specific user, and those images, videos, and other assets determined to be specific to that individual user and have particular significance and / or aesthetic appeal can be identified as best assets for that user. These best assets can be promoted within the media library view and / or within other applications displaying the media library's assets. In this way, the implementation described herein provides an enhanced user experience, making access to the best assets in the media library easier.

[0022] Figure 1 This is a block diagram of an exemplary system 100 for curating assets in a media library. System 100 may include multiple user devices, such as smartphones 102, laptops 104, etc. Each user device 102, 104 may include a media viewing application 108 configured to display assets (such as assets locally stored to a particular user device) that the media viewing application 108 is running on. Furthermore, in one example, any of user devices 102, 104 may be configured to connect to a media server 110 to allow the media viewing application 108 to access additional assets remotely stored to the media server 110 (and possibly not local assets to a particular user device). In one approach, the media viewing application 108 may display remotely and / or locally stored assets in a graphical user interface (GUI) on user devices 102, 104. In another approach, assets may be synchronized across multiple user devices 102, 104, with or without the media server 110.

[0023] Any type of user equipment (including) Figure 1 Those not specifically shown in the text may be included in system 100, such as desktop computers, media devices such as set-top boxes (STBs), mobile phones, digital streaming media devices, smart TVs (TVs), tablet computers, wearable devices such as smartwatches, smart home speakers with displays, digital photo frames, etc.

[0024] Although not shown, a media library can be configured to store, manage, and display multiple assets. These assets can be accessed by users interacting with the media library or by applications configured to access assets from the media library. Any type of asset can be displayed in the media library, such as images, photos, videos, animated images, composite presentations, etc.

[0025] As used herein, a animated image describes a container or other file format comprising a series of images that are manipulated or processed to appear as a coherent moving image when opened or played. In both approaches, the animated image may play automatically at any time it is displayed, or its playback may be activated by some input. Some exemplary animated images include, but are not limited to, Graphics Interchange Format (GIF) files, Portable Web Graphics (PNG) files, Multiimage Web Graphics (MNG) files, and Live Photos. TM wait.

[0026] As used herein, a composite presentation describes the arrangement of media assets, such as images and / or videos, selected according to a theme or purpose, and may be set to audio or music. A composite presentation may also include relevant information such as graphics, maps, information about people identified in the assets, and detailed information about the assets included in the composite presentation. For example, a composite presentation might involve a weekend ski trip and could include videos and photos of the trip, such as pictures of family members drinking hot cocoa around a fire, followed by images of the ski resort, photos of children in ski gear, and videos of family members skiing on a slope. Of course, the order of assets, the music and other audio played, titles, text, or audio descriptions, etc., can be set or modified according to the needs of the user of the application generating the composite presentation.

[0027] Media viewing application 108 can be configured to display assets accessed from a media library via user devices 102, 104. Media viewing application 108 can be any type of application, program, module, instruction set, operating system, firmware, etc., configured to display pictures, images, videos, and / or other media assets captured by media viewing application 108 and accessed from local storage on user devices 102, 104, remote storage on media server 110, and / or from some other data source. Some exemplary media viewing applications include, but are not limited to, online social media platforms, camera modules for mobile phones or smartphones, video recording programs, etc.

[0028] In one example, the media viewing application 108 could be a social media application installed on smartphone 102, which could present a photo set to the user based on shared characteristics or traits found in the photos, such as the location where the photos were captured. This photo set could be stored in a media library, from which the social media application could access the photos for presentation within its GUI on the smartphone 102's display (e.g., in a timeline or calendar view). In another example, the user could be presented with the option to allow the social media application to present the photo set before the social media application displays photos alongside other users and / or devices. It would be beneficial to provide the social media application with only the best photos from the media library for display, based on specific selection criteria. These selection criteria could indicate dates, holidays, travel, themes, moods, scenes, people, places, and / or objects to appear in the selected assets.

[0029] See you again Figure 1 In one method, user equipment 102, 104 can utilize network 106 to access media server 110 or any other remotely accessible data source. Any type of network 106 can be used, such as the Internet, wide area network (WAN), local area network (LAN), wireless local area network (WLAN), virtual private network (VPN), mobile broadband network, etc. Furthermore, in one method, more than one network can be used to connect user equipment 102, 104 to any other system, device, or network.

[0030] Media server 110 can be any type of system or device configured to store assets and provide access to and / or transfer such assets to requesting user devices 102, 104. In one example, media server 110 can be a cloud server hosting images, videos, and other assets, allowing user devices 102, 104 to access these assets as needed. In another approach, media viewing application 108 can be a client application that relies on media server 110 to provide instructions and / or assets for display on user devices 102, 104.

[0031] In one approach, each of the various user devices 102, 104 can be utilized by a single user, and / or the user can input credentials that allow identification of the user when using the shared device. These credentials can be passively input, such as through the use of a user-specific device, or explicitly entered, such as username / password combinations, biometric signatures, token exchanges, etc.

[0032] In one approach, because user usage and interaction with assets in the media library can be tracked across all different user devices 102, 104 that a user can use to access multiple assets in the media library, a general coherent description of how a user interacts with multiple assets in the media library can be generated.

[0033] Figure 2 An exemplary system 200 for tracking user interactions with assets 202 of a media library 206 is shown. As shown, a smartphone 102 is running a media viewing application 108. Over a period of time, the media viewing application 108 may display many different assets 210 from the media library 206 and allow interaction with them. However, for example, just because a certain photo 204 (e.g., a photo of a cat) is displayed on the smartphone 102's screen does not necessarily mean that the user actually likes photo 204. The user may not like cats, or the user may not like this particular cat, or the lighting in the photo may be poor, making it difficult to see the cat. There are many different factors regarding how a user perceives photos or other assets, and any of these factors may be considered when generating a coherent depiction of the user's interactions with the assets 210 of the media library 206. Furthermore, the exemplary system 200 is not limited to use with the smartphone 102, and any computing device can be used in the exemplary system 200, such as a desktop computer, media device such as an STB, digital streaming media device, mobile phone, smart TV, tablet computer, wearable device such as a smartwatch, smart home speaker with display, digital photo frame, etc.

[0034] In one example, each time a user performs some interaction with an asset in media library 206 (such as a photo of a cat 204), that asset is stored in a user-specific profile 208. In one approach, user profile 208 is created by media viewing application 108 and stored locally on smartphone 102. In other approaches, different instances of media viewing application 108 may share and / or synchronize user profile 208 for a specific user who interacts with assets 202 in media library 206 using multiple user devices. In one approach, this sharing or synchronization occurs only after permission is received from the user. This allows all user interactions with assets 202 in media library 206 to be aggregated and used to determine a user's interest in a particular asset, rather than relying on interactions performed on a single device and calculating user interest only for a single device.

[0035] User profile 208 may include information about each interaction between the user and each asset 202 of media library 206 (e.g., assets 204 and 210 displayed on the display of smartphone 102). For example, entries for asset 1, asset 2, asset 3, asset 4, ..., asset N are displayed in user profile 208. Some information is stored in each entry for an asset accessed by the user of smartphone 102. In one example, in one approach, interactions may be tracked over a specific time period, and the tracked interactions may become invalid once that time period has passed. In another approach, values ​​may be assigned to these interactions in the form of interaction scores. Interaction scores may be calculated as described herein or using any other suitable method, and may involve nonlinear relationships including sigmoid and other exponential or higher-order polynomial functions. In yet another approach, the assigned values ​​may decay over time based on the number of years the tracked interactions have been. Again, each asset 202 in media library 206 may have entries, and for assets that have not been accessed or have not yet been accessed in a particular recent time period, these entries may be blank or contain little or no interaction information.

[0036] Some exemplary information stored in entries in the user profile 208 may include, but is not limited to, the number of times the user views an asset on the smartphone 102, the number of times the user shares an asset with one or more other people or devices, the number of times the user plays an asset that can be played (e.g., videos, animated images, montages, etc.), the number of times the user indicates liking an asset and / or marks an asset as a favorite on one or more applications, platforms, etc., the number of times substantially similar assets are stored in the media library 206, timestamps indicating one or more recent interactions with an asset, indications of special settings made to the image capture device for acquiring the asset (e.g., camera settings for capturing photos, special photo types such as panoramas or slow motion, changing the appearance of a photo with filters before capture, etc.), and post-capture editing performed on the asset (e.g., looping, bouncing, long exposure, post-capture filter effects, etc.).

[0037] For example, each entry in user profile 208 includes the name or identifier of the associated asset, such as asset 1, the number of times the user views asset 1 (e.g., view (2)), the number of times the user shares asset 1 (e.g., share (5)), the number of times the user plays asset 1 (e.g., play (0)), and the number of times asset 1 is marked as favorited (e.g., favorite (3)).

[0038] User profile 208 can be used to generate an interaction score for each asset that has an entry in user profile 208. The corresponding interaction score can be based on the sum of a single contribution value assigned to each interaction type. Furthermore, weights can be assigned to increase or decrease the impact that any type of interaction might have on the interaction score. In one approach, all weights can be set to be equal, such that each interaction type provides the same impact on the overall interaction score.

[0039] According to this method, when calculating the interaction score of an asset, certain types of interactions may be weighted higher than other types. For example, sharing may be weighted as having twice the impact on the interaction score as viewing, and playing an asset may contribute three times as much to the interaction score as collecting.

[0040] For example, with a weighted average, asset 1 could have an interaction score equal to the sum of the number of times it has been viewed, shared, played, and favorited (e.g., 2+5+0+3=10), indicating that the user has interacted with asset 1 ten times in different ways. Similarly, asset 2 could have an interaction score of 0, indicating that the user has not yet interacted with asset 2. In these examples, the higher the score, the more interactions have occurred with that particular asset.

[0041] In one approach, interaction scores can be normalized to a scale of 0 to 1 based on the highest computed score. To achieve normalization, each interaction score is divided by the highest computed score, such that the highest computed score becomes 1.0, and all other interaction scores are between 0 and 1 (inclusive). For example, asset 1 would have a highest interaction score of 10, normalized to 1.0. Asset 3 would have a normalized interaction score of (1+2+6+0) / 10 = 0.9, while asset 4 would have a normalized interaction score of 1 / 10 = 0.1.

[0042] Normalization can also be achieved through other calculations, such as feature scaling, average normalization, etc. In one example, the normalization scheme can use the first calculation method, where the normalized score = score - mean / (maximum value - minimum value). Another example is the normalized score = score - mean / standard deviation. According to another example, the normalized score can be obtained based on a global score calculated from other users' scores.

[0043] In one approach, when the number of times an asset is played is weighted at twice the influence of other criteria, asset 3 will have an interaction score of 1 + 2 + 6 * 2 = 15. Therefore, using this weighting scheme, asset 3 will have a higher interaction score than asset 1, which has been tracked to 0 plays.

[0044] Any scheme, algorithm, calculation, or method may be used to generate the interaction score of asset 202 of media library 206, as will be apparent to those skilled in the art, and the calculation, determination, and / or generation of the interaction score is not limited to the explicit description provided herein.

[0045] Once the interaction scores of assets 202 in media library 206 are determined, a subset of assets from media library 206 can be determined based on these interaction scores. For interaction scores normalized to a 1.0 scale, the asset subset may include those assets that achieve interaction scores greater than a threshold score (such as 0.5, 0.7, 0.75, 0.8, 0.9, etc.). In another approach, the asset subset may have a predetermined size (e.g., the total number or percentage of all assets), and the assets with the highest interaction scores populate the asset subset to achieve the predetermined size.

[0046] There can be many different reasons why a particular user might like a particular photo, video, or other asset. Some reasons why a particular user might like an asset could be related to its aesthetic appeal, which can be characterized by pleasure and beauty, such as stunning scenery, artistic intent, beautiful faces and people, etc.

[0047] These reasons why a particular user might like an asset are likely globally consistent for most users, since the concept of aesthetic appeal based on the characteristics and properties of one or more objects of an asset can be approximated. In one example, the aesthetic appeal of an asset can be estimated based on a global aesthetic created by experts on objects of aesthetic appeal, making it possible to generate an aesthetic score for each individual asset in media library 206, as discussed in more detail below.

[0048] Further reasons why specific users might like assets could be related to the emotions and feelings that users attach to the assets based on the content displayed, such as images of great figures (e.g., images of celebrities that users like, alone or with others, relatives, friends, partners, children, deceased relatives, etc.), places of significance (e.g., places users want to visit, places they have visited, places they currently live, places they previously lived, places where friends or relatives lived, etc.), pets or other animals that are meaningful to users, and objects that are significant to users (animal plush toys, sports teams, and souvenirs, etc.). Further reasons why specific users might like assets could be related to lifestyle or activity patterns, such as activities that users enjoy (sports, travel, adventure, road trips, hiking, camping, etc.), hobbies (crafts, creative events, etc.). Still other reasons why specific users might like assets could be related to intellectual stimulation, curiosity, and a thirst for knowledge, such as landmarks, cultural locations and experiences (e.g., culturally specific events such as the Running of the Bulls, May 5th Festival, Lantern Festival, Spring Festival, etc.), religious or cultural festivals and events, etc.

[0049] The reasons why a particular user likes an asset are not typically consistent across all users. For example, one user's relatives may be different from another user's relatives. Similarly, a place one user has visited and photos of that place may be completely irrelevant to a second user, or it might even trigger a negative reaction if the second user had a bad experience there. Therefore, these reasons why a particular user likes an asset are semantically based; for example, a user's liking of an asset is based on the meaning one or more objects within the asset have for the user, rather than necessarily on how one or more objects are depicted within the asset.

[0050] Based on the above discussion, there are two methods for estimating whether a particular user is likely to like a particular asset: aesthetic appeal and semantic appeal. An implementation scheme for determining aesthetic appeal will be described later. An implementation scheme for determining semantic appeal is described below.

[0051] Figure 3An exemplary system 300 is shown for creating a semantic map 310 to predict which specific assets a particular user will like to view in a media library 206. The semantic map 310 may include objects, scenes, and / or characters identified from a subset 302 of assets in the media library 206, which appear in a specific percentage of the assets from the subset 302. This specific percentage may be predetermined and static, or dynamic to allow adjustment to ensure that a robust dataset is represented in a semantic map that includes at least a specific number (e.g., 10, 25, 50, 100, 200, 500, etc.) of the identified features.

[0052] In one approach, asset subset 302 can be determined based on which assets achieve the highest interaction score as described above. When this approach is used, asset subset 302 may be referred to as the "golden set." In another approach, asset subset 302 can be randomly selected from all assets 202 of the media library 206. In one approach, assets included in the asset subset can be selected based on which assets the user has recently interacted with. As those skilled in the art will understand upon reading this description, other methods can be used to determine how many assets and which assets should be included in asset subset 302.

[0053] If a specific object, scene, and / or person cannot be identified from one or more assets, that specific object, scene, and / or person may not be usable to determine the semantic appeal of that object in other assets. Therefore, image analysis is performed on asset subset 302 to identify objects, scenes, and / or people appearing in the assets. Any type of known image analysis that can identify asset features can be used to extract the various objects, scenes, and / or people represented in asset subset 302.

[0054] Some exemplary objects that can be identified in asset subset 302 include, but are not limited to, furniture, toys, food, sports equipment, tools, books, architectural elements (e.g., doors, windows, houses, buildings, columns, etc.), vehicles, celestial and planetary bodies (e.g., the sun, clouds, rain, snow, the earth, stars, comets, etc.), natural elements (e.g., trees, grass, flowers, plants, waterfalls, waves, etc.).

[0055] Some exemplary scenes and locations that can be identified in asset subset 302 include, but are not limited to, beaches, forests, oceans, rivers, urban landscapes, concert and performance venues, indoor spaces, professional and amateur sports venues, commercial or office spaces, and residences.

[0056] Exemplary individuals that can be identified in asset subset 302 include, but are not limited to, family members, friends, partners and colleagues, celebrities, dignitaries, politicians, mentors, historical figures, and deceased relatives.

[0057] To aid in understanding the semantic mapping 310, some assets from a subset 302, representing the highest interaction scores for all assets 202 in the media library 206, are displayed in the media viewing application 108 on the smartphone 102, along with other assets 312 not specifically discussed. In this example, asset 304 (e.g., a beach video) has an interaction score of 28, asset 306 (e.g., a forest video) has an interaction score of 20, asset 308 (e.g., a nighttime photo) has an interaction score of 17, and asset 204 (e.g., a photo of a cat) has an interaction score of 16. These interaction scores are the sum of all views, shares, plays, and favorites tracked for each of these assets (e.g., within a specific recent time period). This recent time period can span any amount of time, such as a day, a week, a month, a year, five years, etc., since the media library was created.

[0058] Based on a high interaction score calculated from multiple interactions and interaction types between the user and the assets, the media viewing application 108 can determine that the user likes each asset in asset subset 302 when the interaction score is used to determine which assets should be included in asset subset 302. However, there is still no coherent understanding of why the user prefers these specific assets in media library 206 over other assets. In order to determine which aspects of these assets are attractive to the user, a semantic map 310 is generated to quantitatively predict and assign the significance and meaning of these assets to the user.

[0059] In one approach, the most frequently occurring features in asset subset 302 are counted to generate a semantic map 310. For example, semantic map 310 shows that "sun" is a feature of 22 different assets in the asset subset (e.g., included in beach video 304 and forest video 306), "nature" is a feature of 18 different assets (e.g., included in beach video 304 and forest video 306), "sky" is a feature of 14 different assets (e.g., included in beach video 304, night photo 308, and forest video 306), and "cat" is a feature of only 1 asset (e.g., included in cat photo 204). Based on the results of organizing which features are most frequently occurring in asset subset 302, semantic map 310 will produce a depiction of which features are most likely to evoke semantic appeal to the user (appealing to the user based on meaning rather than aesthetics).

[0060] For example, "sun" is represented in 22 assets, while "cat" is represented in only 1 asset. This strongly suggests that users do not prefer images of cats, but rather images of the sun. To quantitatively represent this trend of users preferring the sun to cats approximately 22 times more than cats among the assets, a semantic score for each asset 202 in the media library 206 can be calculated based on semantic mapping 310.

[0061] In another approach, the value assigned to a feature used to calculate a semantic score can be based on the individual interaction score of the asset that includes the corresponding feature. For example, for the feature "cat," the interaction score of the only asset including this feature is 16. Therefore, the semantic value assigned to the "cat" feature for calculating the semantic scores of other assets can be based on 16, such as a normalized value, for example, 0.16. When a feature is included in more than one asset, the interaction scores corresponding to all these assets can be summed together to form the semantic value of the feature. In this way, the semantic value of the feature is based on the interaction scores of the assets in asset subset 302, rather than simply on the number of assets that include the feature. This provides that features included in the most interacting assets will receive a higher semantic score than features included in a larger number of assets, even though users typically interact with these assets less.

[0062] The semantic score calculated using the aforementioned techniques can be normalized using the number of assets that include that feature. For example, if a user interacts extensively with pictures of cats but only has a few cat pictures in their media library, whereas if the user interacts equally with pictures of dogs but has many dog ​​pictures in their library, the emphasis (weight) on the feature "cat" is likely to be greater than the emphasis (weight) on the feature "dog". Furthermore, because discovering hard-to-find assets is a beneficial effect of the techniques described in this paper, cat pictures are harder to find in a media library than dog pictures, which are far more numerous. This is another reason for increasing the interaction value assigned to the feature "cat" to exceed that of the feature "dog".

[0063] In one example, if a user has interacted with an asset depicting a “cat” 1000 times and another asset depicting a “dog” 100 times, and both assets are included in asset subset 302, then in this example, semantic map 310 may indicate that the semantic score assigned to the asset depicting the “cat” will be much greater (e.g., twice, five times, or ten times) than the semantic score assigned to the asset depicting the “dog”.

[0064] Once the counts or instances are established, some, all, or specific percentages of the most frequently used features can be determined, and values ​​can be assigned to these features to create semantic scores for the assets in the media library 206.

[0065] To calculate a single semantic score for each asset 202, the corresponding value can be associated with some or all of the features in the semantic map 310. Then, when analyzing a particular asset, for each feature identified in that asset, the corresponding values ​​of the identified features can be summed together to form the semantic score for that asset. In one approach, if a feature such as “cat” is not adequately represented in the subset of assets 302, that feature can be ignored when calculating the semantic scores for other assets.

[0066] In one approach, the feature may be considered in the semantic score calculation in response to a threshold percentage (e.g., 5%, 10%, etc.) of the asset that includes the feature; otherwise, the feature may be ignored in the semantic score calculation.

[0067] In one example, a photo of a cat lazily standing backlit in bright sunlight could be calculated to have a semantic score of 1 + 22 = 23, with the cat at 1 and the sun at 22. Another video showing a landscape of tall trees swaying in a breeze against a beautiful sunset could have a semantic score of 18 + 14 + 22 = 54, with nature at 18, the sky at 14, and the sun at 22. In one approach, the semantic score 23 of the photo of the cat lazily standing and the semantic score 54 of the video of the swaying trees could be compared with one or more other calculated semantic scores of other assets to determine which assets would be most likely to be preferred by users. In this example, it could be determined that users prefer the video of the swaying trees to the photo of the cat lazily standing.

[0068] For example, the value of each feature can be normalized based on the total number of assets analyzed for that feature. Assuming N = 100 indicates that there are 100 total assets in asset subset 302, then "sun" represents 22% or 0.22 of the assets, "nature" represents 18% or 0.18, "sky" represents 14% or 0.14, and "cat" represents 1% or 0.01. In other words, for any feature M appearing in the Y distinct assets of the analyzed asset subset 302, the normalized value can be equal to Y / N. These normalized values ​​can then be used to calculate the semantic score of any asset 202 in the media library that is determined to have the corresponding feature represented therein.

[0069] When semantic mapping 310 is generated, it can be used to analyze multiple assets 202 of media library 206 to generate multiple semantic scores. Based on semantic mapping 310, a semantic score representing and predicting the degree to which the asset is meaningful to the user can be assigned to each asset 202 of media library 206.

[0070] In one approach, the value calculated for each feature in semantic map 310 can be used to determine which assets 202 in media library 206 share specific identified features, and a semantic score corresponding to each asset 202 can be calculated based on semantic map 310 using any of the techniques described above. In addition to those assets calculated from asset subset 302, these semantic scores can also predict which assets among assets 202 will be most likely to be preferred by users.

[0071] Semantic mapping generation can be performed periodically, such as weekly, monthly, or in response to specific triggering mechanisms or conditions. Semantic scores for individual assets can be calculated periodically, such as daily, weekly, or monthly, or in response to one or more new assets being stored in media library 206 and / or one or more assets being removed from media library 206. Furthermore, these calculations can be performed during off-peak usage periods (such as during normal user sleep times (e.g., at night), when user devices are plugged in and not in use, etc.).

[0072] In one example, interaction with an asset can also trigger a recalculation of the interaction score because additional interactions alter the semantic map 310 and aesthetic preferences of the user. Therefore, a recalculation can be triggered in response to: 1) specific changes to the subset of assets used to determine semantic map 310, 2) the total number of interactions that change for a specific percentage of asset subset 302, 3) additional assets added to asset subset 302 (assuming a specific amount of interaction qualifies whether an asset should belong to asset subset 302), etc.

[0073] In one approach, multiple individual aesthetic scores can be generated. Some or all of the assets 202 in the media library 206 can generate corresponding individual aesthetic scores, and these individual aesthetic scores are associated with the assets. The individual aesthetic scores are configured to capture the aesthetic appeal of a particular asset (whether that particular asset is an image, video, animated image, etc.) when compared to a standard considered to be aesthetically pleasing to the general public (e.g., global aesthetics).

[0074] Global aesthetics uses an algorithm or set of algorithms to describe the highest possible aesthetic score (e.g., 1.0) that attempts to determine whether visual aspects and cues in an asset (e.g., lighting, contrast, element positioning, element numbering, etc.) are pleasing to the user. Global aesthetics represent the most aesthetically pleasing image or series of images (in the case of video) as determined by experts in the domain. Global aesthetics can be compared to individual aesthetic scores calculated for a single asset to determine the gap between a particular asset's rating and a global standard, e.g., deviation from global aesthetics. In another approach, the aesthetic score corresponding to the highest-rated asset in media library 206 can be used to normalize all other assets 202 in media library 206 to provide individual aesthetic scores for asset 202. The same algorithm or set of algorithms used to determine global aesthetics is used to individually calculate the aesthetic score for each asset in asset 202. The difference between the global aesthetics and individual aesthetics for each asset is then determined, or an individual aesthetic score is calculated for the entire media library 206 based on the normalization method.

[0075] One approach considers only assets whose personal aesthetic scores meet specific thresholds (e.g., 0.25, 0.4, 0.5, etc. on a 1.0 scale) for further processing, excluding less aesthetically pleasing assets. In other approaches, personal aesthetic scores can be combined with corresponding semantic scores for weighted aggregate analysis of specific assets.

[0076] Weighted aggregate analysis can assign specific weights to semantic scores and specific weights to aesthetic scores, and then take a weighted average of these scores to achieve a weighted semantic / aesthetic score for a specific asset. As is known to those skilled in the art, the specific weights for these scores can be determined in any manner.

[0077] In addition to semantic and personal aesthetic scores, interaction scores for specific assets can be used in the calculation for an overall analysis of which assets users would prefer to see. It should be noted that a subset of assets 302 is analyzed and ultimately helps in selecting which assets to present to the user.

[0078] Furthermore, in one example, a global semantic score can be added to the calculation for overall analysis of what a user would expect to see in a particular asset. For instance, assets acquired or captured during travel, landmarks, weddings, concerts, etc., could have an assigned global semantic score, regardless of whether the user interacts with these assets, because they include certain features known to be favored by that user. This allows for analysis of new assets recently added to the media library 206, even if the user has not yet interacted with them.

[0079] Once the top-rated or best assets in the media library 206 are identified, these top-rated assets can be used by other applications, programs, modules, etc., and will be prominently displayed on the user interface when viewing the assets in the media library 206, rather than simply displaying a portion of all assets 202 in the media library 206.

[0080] In one example, an overemphasized feature can be excluded from the semantic score calculation. For instance, if most assets in the first tier include common features such as “mother,” “sky,” or “nature,” or some other fairly common feature, that feature can be excluded from the semantic map, and a second round of semantic score calculation can be performed to remove the influence of that particular feature that might distort the asset rating.

[0081] Furthermore, diversity can be introduced into Level 1 assets by detecting over-presented features and reducing the impact of including such features on achieving high semantic scores. More diverse features can be included in Level 1 assets when fewer images include specific features. Similarly, when calculating semantic scores, precisely repeated assets and substantially similar (e.g., similar in capture time and content) but not necessarily precisely repeated assets can be removed from consideration. Thus, repetitive and nearly repetitive assets will not be presented in Level 1. However, the presence of repetitive and substantially similar assets can indicate significant user interest in the objects associated with these assets, and semantic scores, interaction scores, and / or personal aesthetic scores can be adjusted to reflect this increased interest.

[0082] Another method, known as discarding, can be used, in which the number of features included in the semantic map is flattened, and features can then be randomly removed from the list. Features that appear more frequently in the asset subset 302 will be most likely to be randomly selected for removal. For example, if there are 700 sunsets and 150 cats, sunsets are most likely to be removed rather than cats. After flattening the semantic map 310, the semantic score can be recalculated. This achieves a balance that prevents overfitting of the model used.

[0083] In one example, another approach to achieving diversification could include calculating the number of assets included in clusters with similar characteristics (e.g., sun, cats, and beaches). It should be noted that assets from smaller clusters should be preferred over those from larger clusters, and this preference can therefore be reflected by assigning higher semantic scores to assets from smaller clusters. For example, given five clusters with corresponding asset counts of 10, 8, 7, 5, and 2, and limited to searching only ten photos out of a total of 32 assets for display, one approach would be to select two assets from each of the five clusters. This approach would be superior to selecting ten assets from the first cluster. This method allows for better control over how assets are selected and discarded; however, the process is also computationally more stringent.

[0084] For example, if a user takes a series of sunset photos on a beach to capture one good shot, and "sun," "beach," "waves," and "landscape" all have high semantic values, it's possible that all the photos in this series would be included in the first level. From a practical point of view, this would be undesirable, as the user wouldn't want to see all the photos in a beach sunset series, but only the best one. Therefore, assets that achieve the highest semantic scores and / or personal aesthetic scores in repetitive and substantially similar asset sets will be considered for selection.

[0085] In one approach, the first-level assets from media library 206 may be those assets rated based on semantic score, personal aesthetic score, or both, within a first percentage of all assets 202. The first percentage may be user-specific or automatically generated and may depend on the number of assets in media library 206 to ensure that not too many or too few assets are presented for display as first-level assets, such as 10%, 8%, 5%, 2%, 1%, etc. Furthermore, in one example, the first percentage may be adjustable.

[0086] Among various methods, the overall analysis can utilize any linear or nonlinear function of interaction scores, semantic scores, global semantic scores, personal aesthetic scores, and / or global aesthetic scores, either alone or in combination with other factors.

[0087] In other methods, a set number of assets may be included in the first level, and / or a number of assets that achieve a predetermined threshold semantic score (e.g., 0.4, 0.5, 0.6, 0.75, 0.8, 0.9, etc. on a 1.0 scale) may be included in the first level of assets.

[0088] According to another example, second-tier high-rated assets from media library 206 can be identified, whose scores are lower than those of first-tier assets. This second tier may include assets rated on a first percentage of all assets 202 based on either semantic score or personal aesthetic score (rather than both). Alternatively, in another approach, these second-tier assets may be rated on both semantic score and personal aesthetic score within a second percentage, or, in a more preferred approach, on only one of the semantic score and personal aesthetic score within a second percentage. The first percentage is smaller than the second percentage, making it more difficult for an asset to be rated within the first percentage and thus considered first-tier.

[0089] Once a second-level asset is identified, in response to the determination that no first-level asset corresponds to the current view of the user interface (which may be based on guidance described below), one or more assets in the second-level assets may be used by the application, program, module, etc. in the current view and prominently displayed in the user interface when viewing the assets of the media library 206.

[0090] The second percentage can be a user-specific, automatically generated, and / or adjustable percentage greater than the first percentage, such as 25%, 20%, 15%, 12%, 10%, etc. Either the first or second percentage can be adjusted to ensure that more or fewer assets are eligible in the first and second level groups for use in the media library 206 and other applications, programs, modules, etc., that can access assets from the media library 206.

[0091] In other methods, a set number of assets may be included in the second level, and / or the number of assets that achieve a predetermined threshold semantic score and personal aesthetic score (e.g., 0.4, 0.5, 0.6, 0.75, 0.8, 0.9, etc. on a 1.0 scale) may be included in the second level of assets.

[0092] In another approach, the application can provide guidance for searching for assets to display before determining the subset of assets to be based on the semantic map. In yet another approach, this guidance can be used to determine which assets are rated highest or best. For example, the guidance could include the date, date range, theme, and / or sentiment that the asset should adhere to. In response to receiving this guidance, when calculating the semantic score for asset 202 in media library 206, the features that best represent the date, date range, theme, and / or sentiment from semantic map 310 can be weighted higher than other features. This allows assets that best adhere to the guidance to have higher semantic scores relative to other assets that do not strictly adhere to the guidance.

[0093] For example, if the guidelines specify a "Christmas" theme, some selected features to be emphasized when calculating the semantic score could include snow, Christmas trees or conifers, Santa Claus, gifts and presents, etc. Based on higher weighting for these features (by assigning them higher semantic values), the highest-rated or best asset will be most likely to include images of these desired features more frequently than other assets. In another approach, the emphasis and weighting of features that do not exemplify Christmas can be removed by lowering the corresponding semantic values ​​of those features. Some examples of features whose semantic values ​​might be lowered to exclude them from the "Christmas" theme include the sun, beaches, other holidays (e.g., July 4th, Halloween, etc.), swimming, baseball, etc.

[0094] For example, if the instruction indicates the emotion of "longing", some selected features to be emphasized when calculating the semantic score may include older family members, significant past events (e.g., birthdays, weddings, graduations, etc.), images of deceased persons that are significant to the user, older images based on timestamps, etc.

[0095] Any theme or emotion that can be converted into a weighted index for semantic score calculation can be used as a guide, such as certain feelings (e.g., happiness, sadness, joy, excitement, etc.), holidays, travel, events, people, etc. The process of applying these themes or emotions to change the semantic score of an asset can vary depending on the specific user, the size of the media library, the type of asset (photos and videos), etc.

[0096] Figure 4An exemplary daily view 400 of a media library application is shown. A media library application is an example of an application, module, software, etc., that can utilize assets from a media library. In daily view 400, aggregate card 412 and daily card 418 are shown as examples. Daily view 400 can be used to display some or all of the assets associated with multiple consecutive dates (e.g., March 12-15, 2019). Daily view 400 also includes a navigation bar 410 for switching between views of the media library application (including, for example, year view 402, month view 404, day view 406, and all assets view 408). Since daily view 406 is the current view, it is highlighted.

[0097] In one approach, in response to determining that the number of assets from any single day of the aggregation period (e.g., March 12-15) is insufficient to display those dates on a separate daycard, day view 400 may display curated assets 416 on aggregate card 412. In other words, the media library application may determine the number of curated assets for each day, compare the number of curated assets for each day to a predefined daycard threshold (e.g., two, three, four, five curated assets, etc.), and determine whether one or more specific dates should be represented by their own daycard or aggregated with other adjacent dates (the previous or next day) to form an aggregate card to represent assets during the aggregation period.

[0098] Based on the determination that the user will prefer key asset 414 over other assets available for display during the aggregation period, the key asset is prominently displayed on aggregation card 412. In one implementation, the determination of key asset 414 may be based on interaction score, semantic score, global semantic score, personal aesthetic score, and / or global aesthetic score corresponding to assets available for display during the aggregation period.

[0099] Similarly, Daycard 418 significantly highlights key asset 420, while key asset 422 is less prominent. In one example, key asset 420 could be identified as a Tier 1 asset based on semantic score and / or personal aesthetic score, while key asset 422 could be identified as a Tier 2 asset based on semantic score and / or personal aesthetic score.

[0100] Figure 5An exemplary monthly view 500 for a media library application is shown. Monthly view 404 is highlighted because it is the current view. This view 500 can be used to display key assets 504, 506 associated with a specific month (e.g., January 2019) on the monthly card 502. In one example, key asset 504 may have a higher semantic score and / or personal aesthetic score than key asset 506, making key asset 504 more prominently displayed. In one approach, all of the key assets 504, 506 are assets from the media library associated with a specific month, and these key assets are determined as the best assets representing that month (e.g., most likely important, desirable, memorable, aesthetically pleasing, etc.) based on higher semantic and / or personal aesthetic scores.

[0101] Key assets can be selected from curatorial assets based on semantic scores and / or personal aesthetic scores of various curatorial assets. In one approach, the curatorial asset that achieves the highest interaction score, semantic score, global semantic score, personal aesthetic score, and / or global aesthetic score for a specific week within a month can be selected as the key asset to represent that week.

[0102] In one approach, each time Monthly Card 502 is displayed, another set of key assets can be selected and displayed from all first- and / or second-tier assets identified for January 2019. In this way, Monthly Card 502 can change dynamically each time it is displayed, but still only displays the key assets for the month in question, which are most likely to be the best assets to acquire from that month.

[0103] The Monthly View 500 also displays key assets 510 related to a specific month (e.g., March 2019) on the Monthly Card 508. These key assets 510 may have similar semantic scores and / or personal aesthetic scores, and are therefore displayed on the display with equal prominence.

[0104] Figure 6 An exemplary year view 600 for a media library application is shown. Year view 402 is highlighted because it is the current view. This view 600 can be used to display one or more key assets 604, 606, 608 associated with a specific year (e.g., 2018) on the year card 602. In one approach, one or more specific months (e.g., April, June, July, December) of a specific year (e.g., 2018) may have key assets 604, 606, 608 for display on the year card 602.

[0105] In one example, key assets 604 and 608 may have higher semantic and / or personal aesthetic scores than key asset 606, making key assets 604 and 608 more prominent than key asset 606. In one approach, all of key assets 604, 606, and 608 are year-related assets from a media library, and these key assets are identified as the best assets representing that year and its individual months (e.g., most likely important, desirable, memorable, aesthetically pleasing, etc.) based on their higher semantic and / or personal aesthetic scores.

[0106] Key assets can be selected from curatorial assets based on the semantic scores and / or personal aesthetic scores of various curatorial assets. In one approach, curatorial assets that achieve the highest interaction score, semantic score, global semantic score, personal aesthetic score, and / or global aesthetic score for a specific month of the year can be selected as key assets to represent that month on the year view 600.

[0107] Annual pass 602 may include other years (e.g., 2019) on another annual pass 610, where key assets 612 for the months of that year (e.g., January, March) are prominently displayed on annual pass 610.

[0108] In one approach, contextual behavior can be used as a factor to determine which assets to display for any of the aforementioned views (day view, aggregate view, month view, and year view). For example, if today is October 31, the theme could be "Halloween," and assets with high semantic score ratings related to Halloween could be selected for display, as previously described.

[0109] Example process

[0110] To enable readers to clearly understand the technical concepts described herein, the following process describes specific steps performed in a particular order. However, one or more steps of a particular process may be rearranged and / or omitted while remaining within the intended scope of the technology disclosed herein. Furthermore, different processes and / or their steps may be combined, recombined, rearranged, omitted, and / or performed in parallel to create different processing flows also within the intended scope of the technology disclosed herein. Moreover, although some details of the technology disclosed herein may be omitted or briefly summarized in the following process for clarity, the details described in the preceding paragraphs can be combined with the process steps described below to obtain a more complete and comprehensive understanding of these processes and the technology disclosed herein.

[0111] Figure 7 This is a flowchart of an exemplary method 700 for determining the significance of assets in a media library. Assets determined using method 700 that will have greater significance for the user can be used as key assets, which are then used in... Figures 4 to 6Displayed in any of the exemplary views described herein or in other views not specifically described herein.

[0112] See you again Figure 7 In operation 702, the computing device can create a semantic map. The semantic map includes features identified from a subset of assets in the media library (e.g., objects, scenes, and characters identified in one example) that appear in a specific percentage or threshold number of assets from the subset. The specific percentage or threshold number of assets can be predetermined and static, or dynamic, to ensure a robust dataset in the semantic map; for example, features with a threshold number (e.g., 10, 25, 50, 100, 200, 500, etc.) can be included in the semantic map. In some methods, the specific percentage can range from about 50% to about 5%, such as 10%. In one method, the specific threshold can be based on the total number of assets in the subset. In another method, features with a predetermined number (e.g., 10, 25, 50, 100, 200, etc.) that appear most frequently in the subset can be included in the semantic map.

[0113] The computing device can be any device capable of processing and analyzing images in a media library, such as mobile phones, laptops, desktop computers, servers, etc.

[0114] In one method, it is possible to... Figure 8 This describes or uses some other methods described herein or known in the art for selecting a subset of assets from a media library that can provide information about which assets are meaningful to a particular user.

[0115] Refer again Figure 7 In operation 704, the computing device may use semantic mapping to analyze one or more assets in the media library to generate multiple semantic scores. Depending on the type or category of the assets or the number of assets to be examined, the number of assets analyzed may include some or all of the media library. A semantic score is generated for each corresponding asset in the media library. This analysis can be performed as described previously.

[0116] In one example, the semantic score for a given asset can be determined based on the number of identified objects, scenes, and people appearing in that asset. In one approach, the semantic score can be normalized in one of several ways described herein or known in the art.

[0117] In another example, a particular feature can be emphasized or de-emphasized when calculating a semantic score by more or less weighting the corresponding semantic values ​​associated with that particular feature.

[0118] In one example, if a user has interacted with an asset depicting a “cat” 1000 times and another asset depicting a “dog” 100 times, and both assets are included in a subset of assets, then in this example, the semantic mapping could indicate that the semantic score assigned to the asset depicting the “cat” will be much greater (e.g., twice, five times, or ten times) than the semantic score assigned to the asset depicting the “dog”.

[0119] In other examples, user interactions with assets within a subset of assets may not be used, or may have less impact, when calculating semantic scores to be assigned to assets using semantic mapping analysis. The analysis of assets results in a single semantic score being assigned to at least some assets within the media library.

[0120] In operation 706, the computing device can determine the identity of a first asset (referred to in some descriptions as a first-level asset) of the media library, which is rated with at least a semantic score within a first percentage of all assets.

[0121] In one example, the first percentage may include a percentage ranging from about 1% to about 10% of total assets, such as about 5% of total assets.

[0122] In operation 708, when viewing assets in the media library, the computing device can prominently display at least one of the first assets in the user interface of the computing device. The user interface may be similar to... Figures 4 to 6 Any of the exemplary views shown or other views not specifically described herein, such as social media interfaces, camera interfaces, video and photo editing interfaces, etc.

[0123] Figure 8 This is a flowchart of an exemplary method 800 for determining a subset of assets in a media library for semantic mapping. In operation 802, a computing device may acquire first information describing a user's interactions with multiple assets in the media library. The first information may be acquired, received, tracked, or otherwise provided to the computing device, directly or indirectly. In one example, user interactions may include any accessing, viewing, playing, sharing, liking, favorited, transferring, or other means indicating that the user has manipulated or connected to a particular asset.

[0124] In one example, the interaction score for each corresponding asset in the media library could be based on the number of times the corresponding asset has been viewed, the number of times the corresponding asset has been played, whether each corresponding asset has been marked as a favorite, the number of times the corresponding asset has been shared with one or more other people, etc. When calculating the interaction score, some or all of these interactions can be considered individually or in combination with other types of interactions.

[0125] Furthermore, in one example, sharing an asset individually or with a smaller group of assets resulted in a higher engagement score compared to sharing it with other assets in a larger group. For instance, sharing a single photo is more indicative of a user's preference for that image than automatically sharing a collection of photos and videos taken during a trip when the user returns home. The former action reflects the user's thought process when choosing a specific photo to share, while the latter might be performed rather perfunctorily, without much thought.

[0126] According to one method, the asset to be viewed can be determined in response to a predetermined amount of time (e.g., 1 second, 2 seconds, 5 seconds, 8 seconds, etc.) during which the corresponding asset is displayed on the user interface of a computing device. In another method, user actions (such as hovering the cursor over the asset, selecting the asset, etc.) can be used to determine when the asset has been viewed.

[0127] In operation 804, the computing device may generate multiple interaction scores based on the first information. A single interaction score is generated for each corresponding asset in the media library. The interaction scores may be calculated according to the foregoing example or according to another method known in the art.

[0128] In operation 806, the computing device may determine which assets among multiple assets in the media library constitute a subset of assets. This determination may be based on the interaction score calculated in operation 804. For example, the asset with the highest interaction score may be included in the subset of assets.

[0129] According to one method, the first information can describe the user's interactions with multiple assets in the media library over a recent period. The time period used to collect the first information can be selected by the user or determined automatically to provide a snapshot of the user's recent activity, such as the past year, past 6 months, past 3 months, past 6 weeks, past 1 month, past 2 weeks, etc.

[0130] In one example, a personal aesthetic score can be generated for some or all of the assets in the media library. A personal aesthetic score is generated for each corresponding asset in the media library. Personal aesthetic scores can be generated based on global aesthetics, and / or any other known method that determines how pleasing or interesting the assets of the media library are to a particular user, as described in the examples herein.

[0131] In one approach, when using personal aesthetic scores, the first asset of the media library can be rated based on both semantic and personal aesthetic scores within a first percentage of all assets.

[0132] In another approach, the computing device may determine second assets in the media library that are rated on a first percentage of all assets for semantic score or personal aesthetic score (but not both), and on a second percentage for both semantic score and personal aesthetic score. After determining second assets in the media library with ratings slightly lower than the first assets, these second assets may be used when the first assets are unavailable. For example, in response to determining that no first asset corresponds to the current view of the user interface, the computing device may prominently display at least one of the second assets in the user interface of the computing device.

[0133] In a specific example, the first percentage may include a range from about 1% to about 10% of all assets in the media library, such as 5% of all assets, and the second percentage may include a range from about 5% to about 25% of all assets in the media library, such as 15% of all assets. In this example, the second percentage is greater than the first percentage; for example, the first percentage is less than the second percentage.

[0134] Figure 9 This is a flowchart of an exemplary method for determining the best assets to display based on a theme. In operation 902, the computing device may acquire first information describing a user's interaction with multiple assets in a media library. The first information may be acquired, received, tracked, or otherwise provided to the computing device, directly or indirectly. In one example, user interaction may include any accessing, viewing, playing, sharing, liking, favorited, transferring, or other means indicating that the user has manipulated or connected to a particular asset.

[0135] The computing device can be any device capable of processing and analyzing images in a media library, such as mobile phones, laptops, desktop computers, servers, etc.

[0136] In one example, the interaction score for each corresponding asset in the media library could be based on the number of times the corresponding asset has been viewed, the number of times the corresponding asset has been played, whether each corresponding asset has been marked as a favorite, the number of times the corresponding asset has been shared with one or more other people, etc. When calculating the interaction score, some or all of these interactions can be considered individually or in combination with other types of interactions.

[0137] In operation 904, the computing device may generate multiple interaction scores based on the first information. A single interaction score is generated for each corresponding asset in the media library. The interaction scores may be calculated according to the foregoing example or according to another method known in the art.

[0138] In operation 906, the computing device may determine which assets among multiple assets in the media library constitute a subset of assets for generating semantic maps. This determination may be based on the interaction score calculated in operation 904. For example, the asset with the highest interaction score may be included in the asset subset.

[0139] According to one method, the first information can describe the user's interactions with multiple assets in the media library over a recent period. The time period used to collect the first information can be selected by the user or determined automatically to provide a snapshot of the user's recent activity, such as the past year, past 6 months, past 3 months, past 6 weeks, past 1 month, past 2 weeks, etc.

[0140] In operation 908, the computing device can create a semantic map based on a topic. The semantic map includes features identified from a subset of assets selected from a media library. Selected features included in the semantic map (e.g., objects, scenes, and characters identified in one example) appear in a specific percentage or threshold number of assets from the asset subset. The specific percentage or threshold number of assets can be predetermined and static, or dynamic, to ensure a robust dataset in the semantic map; for example, features with a threshold number (e.g., 10, 25, 50, 100, 200, 500, etc.) can be included in the semantic map. In some methods, the specific percentage can range from about 50% to about 5%, such as 10%. In one method, the specific threshold can be based on the total number of assets in the asset subset. In another method, features with a predetermined number (e.g., 10, 25, 50, 100, 200, etc.) that appear most frequently in the asset subset can be included in the semantic map.

[0141] Topics can be received by the computing device via user input and can be automatically selected based on data or information available to the computing device, such as the current date, current time, or user sentiment based on recent interactions with other applications. Topics can be used to customize features included in the semantic map and how the inclusion or exclusion of such features affects the semantic score of a particular asset.

[0142] For example, if the theme is "happiness," images from funerals, natural disasters, or other depressing events will not achieve high semantic scores, even if these images adequately represent other features most common in the asset subset. In contrast, images of parties, smiles, and dancing can receive higher semantic scores, provided that these images adequately represent other features most common in the asset subset.

[0143] In Operation 910, the computing device can generate multiple individual aesthetic scores corresponding to some or all of the assets in the media library. Depending on the desired theme, individual aesthetic scores can be generated for the highest-rated asset set based on semantic scores, the asset subset, or other asset groups (such as those assets that are excluded from display or at the threshold used) that can be better analyzed using individual aesthetic scores.

[0144] A personal aesthetic score is generated for each corresponding asset in the media library. Personal aesthetic scores can be generated using any other known method, based on global aesthetics and / or the desired theme, depending on how pleasing or interesting the assets of the media library are to a particular user.

[0145] In Operation 912, the computing device can use semantic mapping to analyze one or more assets in a media library to generate multiple semantic scores. Depending on the type or category of the assets or the number of assets to be examined, the number of assets analyzed may include some or all of the media library. A semantic score is generated for each corresponding asset in the media library. This analysis can be performed as previously described, where the theme is a factor in how various assets are scored. For example, assets with content most closely aligned with the theme will receive a higher semantic score than assets with content unrelated to the theme. Similarly, assets with negative or contrary-to-theme content (e.g., sad content versus a happy theme, landscape photos versus a family theme) may also receive higher scores.

[0146] In one example, the semantic score for a given asset can be determined based on the number of identified objects, scenes, and people appearing in that asset. In one approach, the semantic score can be normalized in one of several ways described herein or known in the art. In another example, a particular feature can be emphasized or de-emphasized when calculating the semantic score by more or less weighting the corresponding semantic values ​​associated with that particular feature.

[0147] In operation 914, the computing device may provide a semantic score and a personal aesthetic score to a second application. The second application may use the semantic score, personal aesthetic score, global semantic score, and / or global aesthetic score corresponding to each asset to determine whether the corresponding asset should be displayed in the second application. This determination may be based on identifying a first asset in the media library (referred to in some descriptions as a first-level asset), which is rated with at least a semantic score (and possibly a personal aesthetic score, global semantic score, and / or global aesthetic score) within a first percentage of all assets.

[0148] In one example, the first percentage could include a percentage ranging from about 1% to about 10% of all assets, such as about 5% of all assets. Alternatively, a desired number of assets could be selected for display, such as if the display has space for a specific number of assets in the GUI, then only those selected numbers of assets could be chosen for display in the GUI.

[0149] Graphical User Interface

[0150] This disclosure describes above various graphical user interfaces (GUIs) for implementing various features, processes, or workflows. These GUIs can be presented on a variety of electronic devices, including, but not limited to, laptop computers, desktop computers, computer terminals, television systems, tablet computers, e-book readers, and smartphones. One or more of these electronic devices may include a touch-sensitive surface. The touch-sensitive surface can process multiple simultaneous input points, including processing data related to the pressure, degree, or position of each input point. Such processing can facilitate gestures using multiple fingers, including pinching and swiping.

[0151] When this disclosure refers to “selecting” a user interface element in a GUI, these terms are understood to include clicking or “hovering” over a user interface element using a mouse or other input device, or touching, tapping, or gesturing over a user interface element using one or more fingers or a stylus. User interface elements can be virtual buttons, menus, selectors, switches, sliders, brushes, knobs, thumbnails, links, icons, radio buttons, checkboxes, and any other mechanisms used to receive input from or provide feedback to the user.

[0152] privacy

[0153] As described above, one aspect of this technology involves collecting and using available data from various sources to provide behavioral curation of assets within a media library to identify key assets within the library. This disclosure contemplates that, in some instances, the collected data may include personal information that uniquely identifies or can be used to contact or locate specific individuals. Such personal information may include demographic data, location-based data, telephone numbers, email addresses, Twitter IDs, home addresses, data or records related to a user's health or fitness level (e.g., vital sign measurements, medication information, exercise information), date of birth, or any other identifying or personal information.

[0154] This disclosure recognizes that the use of such personal information data in this technology can benefit users. For example, personal information data can be used to provide behavioral curation of assets to identify key assets in a media library. Furthermore, this disclosure also anticipates other uses of personal information data that benefit users. For example, health and fitness data can be used to provide insights into a user's overall health status or as positive feedback for individuals using technology to pursue health goals.

[0155] This disclosure assumes that entities responsible for collecting, analyzing, disclosing, transmitting, storing, or otherwise using such personal information data will comply with established privacy policies and / or privacy practices. Specifically, such entities should implement and adhere to privacy policies and practices that are generally recognized as meeting or exceeding industry or governmental requirements for maintaining the privacy and security of personal information data. Such policies should be easily accessible to users and should be updated as data collection and / or use change. Personal information from users should be collected for the entity's lawful and reasonable purposes and not shared or sold outside of these lawful uses. Furthermore, such collection / sharing should be conducted only after obtaining informed consent from users. In addition, such entities should consider taking any necessary steps to protect and safeguard access to such personal information data and ensure that others with access to such personal information data comply with their privacy policies and processes. Additionally, such entities may subject themselves to third-party assessments to demonstrate their compliance with widely accepted privacy policies and practices. Furthermore, policies and practices should be adapted to the specific types of personal information data collected and / or accessed, and to applicable laws and standards, including specific considerations regarding jurisdiction. For example, in the United States, the collection or acquisition of certain health data may be governed by federal and / or state laws, such as the Health Insurance Portability and Accountability Act (HIPAA); while in other countries, health data may be subject to other regulations and policies and should be handled accordingly. Therefore, different privacy practices should be maintained for different types of personal data in each country.

[0156] Regardless of the foregoing, this disclosure also anticipates implementation schemes for users to selectively block the use or access to personal information data. That is, this disclosure anticipates providing hardware and / or software components to prevent or block access to such personal information data. For example, when providing behavioral curation, this technology can be configured to allow users to opt-in or opt-out at any time during or after registering for the service, choosing to participate in the collection of personal information data. As another example, users can choose not to provide emotion-related data for a targeted content delivery service. In another example, users can choose to limit the length of time emotion-related data is retained, or completely prohibit the development of baseline emotion profiles that can be used to guide the prominent display of certain assets in a media library. In addition to providing opt-in and opt-out options, this disclosure envisions providing notifications related to access to or use of personal information. For example, users can be notified when downloading an application that their personal information data will be accessed, and then reminded again just before the application accesses the personal information data.

[0157] Furthermore, the purpose of this disclosure is to manage and process personal information data to minimize the risk of unintentional or unauthorized access or use. Once data is no longer needed, this risk can be minimized by limiting data collection and deleting data. Additionally, and where applicable, including in certain health-related applications, data deidentification can be used to protect user privacy. Deidentification can be facilitated, where appropriate, by removing specific identifiers (e.g., date of birth, etc.), controlling the amount or specificity of stored data (e.g., collecting location data at the city level rather than the address level), controlling how data is stored (e.g., aggregating data among users), and / or other methods.

[0158] Therefore, while this disclosure broadly covers the use of personal information data to implement one or more of the various disclosed embodiments, it is also contemplated that various embodiments can be implemented without access to such personal information data. That is, various embodiments of the present invention will not be rendered inoperable due to the absence of all or part of such personal information data. For example, preferences can be inferred based on non-personal information data or an absolute minimum amount of personal information (such as content being requested by a user's associated device, other non-personal information available for the curation process, or publicly available information), thereby selecting and presenting specific assets of a media library to the user.

[0159] Exemplary System Architecture

[0160] Figure 10 To be achievable Figures 1 to 9 A block diagram of an exemplary computing device 1000 illustrating its features and processes. The computing device 1000 may include a memory interface 1002, one or more data processors, a graphics processor and / or a central processing unit 1004, and a peripheral device interface 1006. The memory interface 1002, the one or more processors 1004, and / or the peripheral device interface 1006 may be separate components or integrated into one or more integrated circuits. The various components in the computing device 1000 may be coupled via one or more communication buses or signal lines.

[0161] Sensors, devices, and subsystems can be coupled to peripheral interface 1006 to facilitate multiple functions. For example, motion sensor 1010, light sensor 1012, and proximity sensor 1014 can be coupled to peripheral interface 1006 to facilitate orientation, illumination, and proximity functions. Other sensors 1016 can also be connected to peripheral interface 1006, such as Global Navigation Satellite System (GNSS) (e.g., GPS receiver), temperature sensors, biometric sensors, magnetometers, or other sensing devices to facilitate related functions.

[0162] The camera subsystem 1020 and optical sensor 1022 (e.g., a charge-coupled device (CCD) or complementary metal-oxide-semiconductor (CMOS) optical sensor) can be used to facilitate camera functions such as recording photos and video clips. The camera subsystem 1020 and optical sensor 1022 (e.g., by performing facial recognition analysis) can be used to collect user images to be used during user authentication.

[0163] Communication functionality can be facilitated by one or more wireless communication subsystems 1024, which may include radio frequency receivers and transmitters and / or optical (e.g., infrared) receivers and transmitters. The specific design and implementation of the communication subsystem 1024 may depend on the computing device 1000's intended communication network via one or more communication networks operating thereon. For example, the computing device 1000 may include subsystems designed for communication via GSM networks, GPRS networks, EDGE networks, Wi-Fi or WiMax networks, and Bluetooth. TM A network-operated communication subsystem 1024. Specifically, the wireless communication subsystem 1024 may include a managed protocol that allows device 100 to be configured as a base station for other wireless devices.

[0164] The audio subsystem 1026 can be coupled to the speaker 1028 and the microphone 1030 to facilitate voice-enabled functions such as speaker recognition, voice copying, digital recording, and telephone functions. The audio subsystem 1026 can be configured to facilitate, for example, processing of voice commands, voiceprint identification, and voice authentication.

[0165] I / O subsystem 1040 may include touch surface controller 1042 and / or one or more other input controllers 1044. Touch surface controller 1042 may be coupled to touch surface 1046. Touch surface 1046 and touch surface controller 1042 may, for example, use any of a variety of touch-sensitive technologies (including but not limited to capacitive, resistive, infrared, and surface acoustic wave technologies) and other proximity sensor arrays or other elements for determining one or more points of contact with touch surface 1046 to detect contact and movement or their interruption.

[0166] One or more other input controllers 1044 may be coupled to other input / control devices 1048, such as one or more buttons, rocker switches, thumbwheels, infrared ports, USB ports, and / or pointing devices such as styluses. One or more buttons (not shown) may include volume up / down buttons for speaker 1028 and / or microphone 1030.

[0167] In one implementation, pressing the button for a first duration unlocks the touch surface 1046; and pressing the button for a second duration longer than the first duration turns the computing device 1000 on or off. Pressing the button for a third duration activates a module for voice control or voice commands, allowing the user to speak commands into the microphone 1030 to have the device execute the commands. The user can customize the function of one or more buttons. For example, the touch surface 1046 can also be used to implement virtual or soft buttons and / or a keyboard.

[0168] In some implementations, the computing device 1000 may display recorded audio and / or video files, such as MP3, AAC, and MPEG files. In some implementations, the computing device 1000 may include an MP3 player (such as an iPod). TM () function.

[0169] Memory interface 1002 may be coupled to memory 1050. Memory 1050 may include high-speed random access memory and / or non-volatile memory, such as one or more disk storage devices, one or more optical storage devices, and / or flash memory (e.g., NAND, NOR). Memory 1050 may store operating system 1052, such as Darwin, RTXC, LINUX, UNIX, OS X, WINDOWS, or embedded operating system such as VxWorks.

[0170] Operating system 1052 may include instructions for handling basic system services and for performing hardware-related tasks. In some implementations, operating system 1052 may be a kernel (e.g., a UNIX kernel). In some implementations, operating system 1052 may include instructions for performing voice authentication. For example, operating system 1052 may implement behavior curation features, as referenced... Figures 1 to 9 describe.

[0171] The memory 1050 may also store communication instructions 1054 to facilitate communication with one or more additional devices, one or more computers, and / or one or more servers. The memory 1050 may include graphical user interface instructions 1056 that facilitate graphical user interface processing; sensor processing instructions 1058 that facilitate sensor-related processing and functions; telephone instructions 1060 that facilitate telephone-related processes and functions; electronic message processing instructions 1062 that facilitate electronic message processing and functions; web browser instructions 1064 that facilitate web browsing-related processes and functions; media processing instructions 1066 that facilitate media processing-related processes and functions; GNSS / navigation instructions 1068 that facilitate GNSS and navigation-related processes and instructions; and / or camera instructions 1070 that facilitate camera-related processes and functions.

[0172] Memory 1050 can store software instructions 1072 to facilitate other processes and functions, such as references Figures 1 to 9 The description of the curatorial process and functions.

[0173] The memory 1050 may also store other software instructions 1074, such as network video instructions that facilitate processes and functions related to network video; and / or network shopping instructions that facilitate processes and functions related to online shopping. In some embodiments, the media processing instructions 1066 are divided into audio processing instructions and video processing instructions to facilitate processes and functions related to audio processing and processes and functions related to video processing, respectively.

[0174] Each of the instructions identified above and in the application corresponds to an instruction set used to perform one or more of the functions described above. These instructions do not need to be implemented as a separate software program, process, or module. The memory 1050 may include additional instructions or fewer instructions. Furthermore, various functions of the computing device 1000 may be implemented in hardware and / or software (including in one or more signal processing and / or application-specific integrated circuits).

Claims

1. A method for behavioral curation of media assets, the method comprising: The computing device acquires initial information describing user interactions with multiple assets in the media library that occurred within a specific time period; The computing device generates multiple interaction scores based on at least a portion of the first information, with each interaction score corresponding to a corresponding asset in the media library, wherein: for each corresponding asset, the interaction score is based at least in part on the number of assets shared with the corresponding asset; The computing device determines a subset of assets from a plurality of assets in the media library, wherein the assets in the subset have corresponding interaction scores that satisfy a threshold interaction score; and When viewing the assets in the media library, at least one asset from the subset of assets is prominently displayed on the user interface of the computing device.

2. The method of claim 1, further comprising: The computing device identifies objects, scenes, and characters appearing in a specific percentage of assets within a subset of the media library's assets; The computing device creates a semantic mapping between the identified objects, scenes, and characters and the corresponding values ​​indicating the representations of the identified objects, scenes, and characters in the asset subset of the media library; The computing device analyzes the specific asset using semantic mappings corresponding to objects, scenes, and characters in each specific asset among multiple assets in the media library to generate multiple semantic scores, each semantic score corresponding to each corresponding asset in the media library; as well as The computing device determines the first asset in the media library with a semantic rating within a first percentage of all assets; Prominently displaying at least one asset in the subset of assets includes prominently displaying at least one of the first assets on the user interface of the computing device when viewing the assets of the media library.

3. The method of claim 2, wherein the semantic score of the corresponding asset is determined based on the number of identified objects, scenes and people appearing in the corresponding asset.

4. The method of claim 2, further comprising: The computing device generates multiple personal aesthetic scores, each personal aesthetic score corresponding to a corresponding asset in the media library; The personal aesthetic rating of the first asset in the aforementioned media library is also within the top percentage of all assets.

5. The method of claim 1, wherein the specific time period corresponds to the most recent time period.

6. The method of claim 1, wherein: For each corresponding asset in the media library, the plurality of interaction scores are based on at least one of the following: a) the number of times the corresponding asset is viewed, b) the number of times the corresponding asset is played, c) whether the corresponding asset is marked as a favorite, and d) the number of times the corresponding asset is shared with one or more other people; as well as In response to the corresponding asset being displayed on the user interface of the computing device for a predetermined amount of time, the corresponding asset is determined to be viewed.

7. The method of claim 1, wherein for each corresponding asset, the interaction score is inversely proportional to the number of assets shared with the corresponding asset.

8. The method of claim 1, wherein obtaining the first information describing the user interaction comprises: The computing device tracks multiple first user interactions occurring in a first application with one or more assets among multiple assets of the media library; The computing device stores the first set of information corresponding to the interaction with the plurality of first users into the user configuration file on the computing device; as well as The computing device removes information from the user profile that corresponds to user interactions that exceed the specified time period.

9. The method of claim 8, wherein obtaining first information describing the user interaction further comprises: The computing device determines multiple second user interactions with one or more assets among a plurality of assets in the media library, the plurality of second user interactions occurring in a second application different from the first application; and The computing device stores the second set of information corresponding to the interaction with the plurality of second users into the user configuration file on the computing device.

10. The method of claim 1, wherein for each corresponding asset, the increase in the interaction score is inversely proportional to the number of assets shared with the corresponding asset.

11. A system for behavioral curation of media assets, the system comprising: One or more processors; and One or more non-transitory computer-readable media, the one or more non-transitory computer-readable media including instructions that, when executed by the one or more processors, cause the one or more processors to perform operations, the operations including: The computing device acquires initial information describing user interactions with multiple assets in the media library that occurred within a specific time period; The computing device generates multiple interaction scores based on at least a portion of the first information, with each interaction score corresponding to a corresponding asset in the media library. For each corresponding asset, the interaction score is based at least in part on the number of assets shared with the corresponding asset. The computing device determines a subset of assets from a plurality of assets in the media library, wherein the assets in the subset have corresponding interaction scores that satisfy a threshold interaction score; and When viewing the assets in the media library, at least one asset from the subset of assets is prominently displayed on the user interface of the computing device.

12. The system of claim 11, wherein: The operation further includes: The computing device identifies objects, scenes, and characters appearing in a specific percentage of assets within a subset of the media library's assets; The computing device creates a semantic mapping between the identified objects, scenes, and characters and the corresponding values ​​indicating the representations of the identified objects, scenes, and characters in the asset subset of the media library; The computing device analyzes the specific asset using semantic mappings corresponding to objects, scenes, and characters in each specific asset of the media library to generate multiple semantic scores, each semantic score corresponding to a corresponding asset in the media library; and The computing device determines the first asset in the media library with a semantic rating within a first percentage of all assets; and Prominently displaying at least one of the asset subsets includes prominently displaying at least one of the first assets on the user interface of the computing device when viewing the assets of the media library.

13. The system of claim 12, wherein the semantic score of the corresponding asset is determined based on the number of identified objects, scenes and people appearing in the corresponding asset.

14. The system of claim 12, wherein: The operation further includes generating multiple personal aesthetic scores by the computing device, each personal aesthetic score corresponding to a corresponding asset in the media library; and The personal aesthetic rating of the first asset in the media library is also within the top percentage of all assets.

15. The system of claim 11, wherein the specific time period corresponds to the most recent time period.

16. The system of claim 11, wherein: For each corresponding asset in the media library, the plurality of interaction scores are based on at least one of the following: a) the number of times the corresponding asset is viewed, b) the number of times the corresponding asset is played, c) whether the corresponding asset is marked as a favorite, and d) the number of times the corresponding asset is shared with one or more other people; as well as In response to the corresponding asset being displayed on the user interface of the computing device for a predetermined amount of time, the corresponding asset is determined to be viewed.

17. The system of claim 11, wherein for each corresponding asset, the interaction score is inversely proportional to the number of assets shared with the corresponding asset.

18. The system of claim 11, wherein obtaining the first information describing the user interaction comprises: The computing device tracks multiple first user interactions occurring in a first application with one or more assets among multiple assets of the media library; The computing device stores the first set of information corresponding to the interaction with the plurality of first users into the user configuration file on the computing device; as well as The computing device removes information from the user profile that corresponds to user interactions that exceed the specified time period.

19. The system of claim 18, wherein obtaining the first information describing the user interaction further comprises: The computing device determines multiple second user interactions with one or more assets among a plurality of assets in the media library, the plurality of second user interactions occurring in a second application different from the first application; and The computing device stores the second set of information corresponding to the interaction with the plurality of second users into the user configuration file on the computing device.

20. The system of claim 11, wherein for each corresponding asset, the increase in the interaction score is inversely proportional to the number of assets shared with the corresponding asset.

Images