Automatic generation of character groups and image-based creations

Abstract

Embodiments described herein relate to methods, devices, and computer-readable media for generating and providing image-based creations. A computer-implemented method includes: obtaining a plurality of segments, each segment associated with a corresponding time period and including a respective set of images and a person identifier for each image. The method further includes forming a respective cluster for each segment that includes at least two person identifiers. The method further includes determining whether one or more person identifiers are included in fewer than a threshold number of clusters and, in response, removing the one or more person identifiers from the clusters that include the one or more person identifiers. The method further includes merging identical clusters to obtain a plurality of groups of persons and providing a user interface that includes an image-based creation based on a particular group of persons, each group of persons containing two or more person identifiers.

Description

Background Art

[0001] Users of devices such as smart phones or other digital cameras capture a large number of photos and videos and store them in their image libraries. Users utilize these libraries to view their photos and videos to recall various events such as birthdays, weddings, vacations, trips, etc. Users may have large image libraries that contain thousands of images taken over a long period of time.

[0002] The background art description provided herein is to generally present the context of the present disclosure. The work of the presently named inventors, to the extent it is described in this background art section, and aspects of the specification that may not otherwise be prior art at the time of filing, are not expressly or implicitly admitted to be prior art against the present disclosure. Summary of the Invention

[0003] Embodiments described herein relate to methods, devices, and computer-readable media for generating and providing image-based creations.

[0004] In some embodiments, a computer-implemented method includes: obtaining a plurality of segments, where each segment is associated with a corresponding time period and includes a corresponding set of images and a person identifier for each image in the corresponding set of images. The method further includes: forming a corresponding cluster for each segment, where each cluster includes at least two person identifiers. The method further includes: determining whether one or more person identifiers are included in fewer than a threshold number of clusters. The method further includes: if it is determined that one or more person identifiers are included in fewer than the threshold number of clusters, removing the one or more person identifiers from the clusters that include the one or more person identifiers. The method further includes: after the removal, merging identical clusters to obtain a plurality of groups of persons, where each group of persons includes two or more person identifiers. The method further includes: providing a user interface that includes an image-based creation based on a particular group of persons among the plurality of groups of persons.

[0005] In some embodiments, forming a corresponding cluster for each segment may include: mapping all person identifiers that appear in at least one image in the set of images of the segment to a cluster; determining the number of images that include the most frequent person identifier in the segment from the set of images of the segment; and removing person identifiers associated with fewer than a threshold number of images from the cluster. In some embodiments, the threshold number of images may be determined based on the number of images that include the most frequent person identifier in the segment.

[0006] In some embodiments, the method may further include: determining that at least one group of persons is associated with fewer than a threshold number of segments. The method may further include: in response to determining that at least one group of persons is associated with fewer than a threshold number of segments, combining the at least one group of persons with one or more other groups of persons. Each of the one or more other groups of persons may include a subset of the person identifiers included in the at least one group of persons. In these embodiments, providing the user interface may be performed after combining the at least one group of persons.

[0007] In some embodiments, combining the same clusters may include: associating a corresponding set of segments with each of a plurality of groups of persons based on the clusters. In some embodiments, providing an image-based creation may include: selecting a subset of images from the corresponding set of images included in the segments associated with a particular group of persons for the image-based creation. Each image in the subset of images may depict a person corresponding to at least two of the two or more person identifiers included in the particular group of persons.

[0008] In some embodiments, providing the user interface is performed in response to at least one of: detecting that an image matching a particular group has been recently captured; detecting that the current date matches a date associated with a particular group of persons; or detecting an event matching a particular group of persons.

[0009] In some embodiments, providing an image-based creation may include: selecting a subset of images for the image-based creation based on a particular group of persons. Each image in the subset of images may depict a person corresponding to at least two of the two or more person identifiers included in the particular group of persons. The method may further include: generating an image-based creation based on the subset of images. Each image in the subset of images may depict a person corresponding to each of the two or more person identifiers included in the particular group of persons. In some embodiments, each image in the subset of images may depict a person corresponding to at least two of the two or more person identifiers included in the particular group of persons. In some embodiments, the subset of images may include images from at least two of a plurality of segments. In some embodiments, the subset of images may be selected such that the subset provides one or more of: location diversity, pose diversity, or visual diversity.

[0010] Some embodiments include a non-transitory computer-readable medium having instructions stored thereon that, when executed by a processor, cause the processor to perform operations that include: obtaining a plurality of segments, where each segment is associated with a corresponding time period and includes a respective set of images and a person identifier for each image in the respective set of images. The operations further include: forming a respective cluster for each segment, where each cluster includes at least two person identifiers. The operations further include: determining whether one or more person identifiers are included in fewer than a threshold number of clusters; and if it is determined that one or more person identifiers are included in fewer than a threshold number of clusters, removing the one or more person identifiers from the clusters that include the one or more person identifiers. The operations further include: merging identical clusters to obtain a plurality of groups of persons, where each group of persons includes two or more person identifiers. The operations further include: providing a user interface that includes an image-based creation based on a particular group of persons among the plurality of groups of persons.

[0011] In some embodiments, the operation of forming a respective cluster for each segment may include: mapping all person identifiers that appear in at least one image in the set of images of the segment to a cluster; determining, from the set of images of the segment, the number of images that include the most frequent person identifier in the segment; and removing from the cluster the person identifiers associated with fewer than a threshold number of images. In some embodiments, the threshold number of images may be determined based on the number of images that include the most frequent person identifier in the segment.

[0012] In some embodiments, the non-transitory computer-readable medium may have further instructions stored thereon that cause the processor to perform operations that include: determining that at least one group of persons is associated with fewer than a threshold number of segments; and in response to determining that at least one group of persons is associated with fewer than a threshold number of segments, combining the at least one group of persons with one or more other groups of persons, where each of the one or more other groups of persons includes a subset of the person identifiers included in the at least one group of persons. In these embodiments, the operation of providing the user interface may be performed after merging the at least one group of persons.

[0013] In some embodiments, the operation of providing an image-based creation may include: selecting a subset of images for the image-based creation based on a particular group of persons, where each image in the subset of images depicts a person corresponding to at least two of the two or more person identifiers included in the particular group of persons; and generating an image-based creation based on the subset of images.

[0014] In some embodiments, the operation of merging identical clusters may include: associating a corresponding set of segments with each of multiple groups of people based on the clusters. In some embodiments, providing an image-based creation may include: selecting a subset of images from the corresponding set of images included in the segments associated with a particular group of people for the image-based creation.

[0015] Some embodiments may include a computing device that includes a processor and a memory coupled to the processor. The memory may store instructions thereon that, when executed by the processor, cause the processor to perform operations that include: obtaining a plurality of segments, where each segment is associated with a corresponding time period and includes a corresponding set of images and a person identifier for each image in the corresponding set of images. The operations further include: forming a corresponding cluster for each segment, where each cluster includes at least two person identifiers. The operations further include: determining whether one or more person identifiers are included in fewer than a threshold number of clusters; and if it is determined that one or more person identifiers are included in fewer than a threshold number of clusters, removing the one or more person identifiers from the clusters that include the one or more person identifiers. The operations may further include: after the removal, merging identical clusters to obtain multiple groups of people, where each group of people includes two or more person identifiers. The operations further include: providing a user interface that includes an image-based creation based on a particular group of people among the multiple groups of people.

[0016] In some embodiments, the operation of forming a corresponding cluster for each segment may include: mapping all person identifiers that appear in at least one image in the set of images of the segment to the cluster; determining the number of images that include the most frequent person identifier in the segment from the set of images of the segment; and removing from the cluster person identifiers associated with fewer than a threshold number of images. In some embodiments, the threshold number of images may be determined based on the number of images that include the most frequent person identifier in the segment.

[0017] In some embodiments, the memory may store further instructions thereon that, when executed by the processor, cause the processor to perform operations that include: determining that at least one group of people is associated with fewer than a threshold number of segments; and in response to determining that at least one group of people is associated with fewer than a threshold number of segments, combining the at least one group of people with one or more other groups of people. In some embodiments, each of the one or more other groups of people may include a subset of the person identifiers included in the at least one group of people. In these embodiments, the operation of providing the user interface may be performed after merging the at least one group of people.

[0018] In some embodiments, operations for providing an image-based creation may include: selecting a subset of images for an image-based creation based on a specific group of persons, where each image in the subset of images depicts a person corresponding to at least two of two or more person identifiers included in the specific group of persons; and generating an image-based creation based on the subset of images.

[0019] In some embodiments, operations for merging identical clusters may include: associating a corresponding set of segments with each of a plurality of groups of persons based on the clusters. In some embodiments, providing an image-based creation may include: selecting a subset of images for an image-based creation from a corresponding set of images included in segments associated with a specific group of persons. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is a block diagram of an example network environment that can be used in one or more embodiments described herein.

[0021] Figure 2 is a flowchart illustrating an example method for providing a user interface including an image-based creation according to some embodiments.

[0022] Figures 3A - 3F Illustrates different stages of obtaining groups of persons from segments in an image library according to some embodiments.

[0023] Figure 4A Illustrates an example user interface according to some embodiments described herein.

[0024] Figure 4B Illustrates an example user interface according to some embodiments.

[0025] Figure 5 is a block diagram of an example computing device that can be used to implement one or more features described herein. DETAILED DESCRIPTION

[0026] The present disclosure relates to automatically determining groups of prominent persons based on a user's image library and providing image-based creations for these groups of persons at appropriate times. For example, a user may have an image library including thousands of photos taken over a long period of time, such as the user's lifetime. Browsing such images can be tedious as the user may encounter a large number of uninteresting or repetitive images. Additionally, although search functions may enable a user to identify images that meet certain criteria (e.g., images captured within a specific time period, images depicting a specific person, etc.), the user needs to explicitly formulate a query and perform a search to view the images.

[0027] In an image library, a subset of the depicted people may be both prominent and important to the user. For example, such a subset can include the user's immediate family members (e.g., spouse, children, parents), pets, close friends, etc. The user is more likely to be interested in images of such a subset of people because these are important people in their lives.

[0028] This document describes techniques for automatically identifying groups of prominent people in a user's image library, e.g., groups of two or more people where each person has a prominent relationship with the user. The techniques are performed under the specific permission of the user to access user data for such automatic identification. In some embodiments, the user image library can be divided into multiple episodes such that each episode can correspond to an event (e.g., a major event such as a wedding or graduation ceremony or a minor event such as a hiking trip). According to the techniques described herein, the episodes in the user image library are analyzed to identify clusters of people associated with each episode. People who appear infrequently or only a few times in the episode are removed from the clusters of people, and any duplicate clusters of people are merged to obtain groups of people.

[0029] In some embodiments, if multiple groups of people are identified, e.g., based on a trigger condition, a subset of these groups of people can be utilized to generate an image-based creation. For example, the larger groups of people rather than the smaller groups of people can be used for the image-based creation. If multiple groups of people of the same size are determined, the groups of people can be ranked based on person identifiers, based on whether the most recent image includes the group of people, based on the relative frequency of each group of people in the user library, etc.

[0030] Based on the identified groups of people, image-based creations such as slideshows, collages, image albums, etc. are automatically generated. A user interface is provided to the user to view the image-based creation. The image-based creation can be generated at an appropriate time (e.g., before and after an upcoming event, at a time when the user periodically browses the images in the image library, etc.) and can be provided via the user interface. Thus, the techniques automatically identify groups of prominent people in the image library and provide a user interface that enables the user to view image-based creations of these groups of people.

[0031] Figure 1 A block diagram illustrating an example network environment 100 that can be used in some embodiments described herein. In some embodiments, the network environment 100 includes one or more server systems, e.g., Figure 1the server system 102 in the example of; and a plurality of client devices, e.g., client devices 120 - 126, each associated with a respective user among users U1 - U4. Each of the server system 102 and the client devices 120 - 126 can be configured to communicate with the network 130.

[0032] The server system 102 can include a server device 104 and a database 106. In some embodiments, the server device 104 can provide an image application 106a. In Figure 1 and the remaining figures, the letter following a reference numeral such as "106a" refers to a reference to the element having that particular reference numeral. A reference numeral such as "106" without the following letter in the text represents a general reference to an embodiment of the element carrying that reference number.

[0033] The image database 110 can be stored on a storage device that is part of the server system 102. In some embodiments, the image database 110 can be implemented using a relational database, a key - value structure, or other types of database structures. In some embodiments, the image database 110 can include a plurality of partitions, each corresponding to a respective image library for each user among users U1 - U4. For example, as Figure 1 seen, the image database 110 can include a first image library (image library 1, 108a) for user U1, and other image libraries (IL - 2, IL - 3,..., IL - n) for various other users. Although Figure 1 a single image database 110 is shown, it can be understood that the image database 110 can be implemented as a distributed database, e.g., on multiple database servers. Additionally, although Figure 1 a plurality of partitions are shown, one partition for each user, in some embodiments, each image library can be implemented as a separate database.

[0034] The image library 108a can store a plurality of images associated with user Ul, metadata associated with the plurality of images, and one or more other database fields stored in association with the plurality of images. Access permissions to the image library 108a can be restricted such that user U1 can control how, for example, the images and other data in the image library 108a are accessed via the image application 106, via other applications, and / or via one or more other users. The server system 102 can be configured to enforce the access permissions such that the image data of a particular user is accessible only with the user's permission.

[0035] The images referred to in this document can include digital images having pixels, where the pixels have one or more pixel values (e.g., color values, luminance values, etc.). The images can be still images (e.g., still photos, images with a single frame, etc.), moving images (e.g., animations, GIF animations, cinematography where part of the image includes movement while other parts are static, etc.), or videos (e.g., a sequence of images or image frames that can optionally include audio). The images as used herein can be understood to be any of the above. For example, the embodiments described herein can be used with still images (e.g., photos or other images), videos, or moving images.

[0036] The network environment 100 can also include one or more client devices, e.g., client devices 120, 122, 124, and 126, which can communicate with each other and / or with the server system 102 via the network 130. The network 130 can be any type of communication network, including the Internet, a local area network (LAN), a wireless network, a switch or hub connection, etc., or a combination thereof. In some embodiments, the network 130 can include peer-to-peer communication between devices, e.g., using a peer-to-peer wireless protocol (e.g., Wi-Fi Direct, etc.). An example of peer-to-peer communication between two client devices 120 and 122 is shown by arrow 132.

[0037] In various embodiments, users U1, U2, U3, and U4 can use the respective client devices 120, 122, 124, and 126 to communicate with the server system 102 and / or with each other. In some examples, users U1, U2, U3, and U4 can interact with each other via applications running on the respective client devices and / or the server system 102 and / or via network services such as social network services or other types of network services implemented on the server system 102. For example, the respective client devices 120, 122, 124, and 126 can communicate data to and from one or more server systems (e.g., the server system 102).

[0038] In some embodiments, the server system 102 can provide appropriate data to the client devices such that each client device can receive communication content or shared content uploaded to the server system 102 and / or the network service. In some examples, users U1 - U4 can interact via image sharing, audio or video conferencing, audio, video, or text chat, or other communication modes or applications.

[0039] The network services implemented by the server system 102 may include systems that allow users to perform various communications, form links and associations, upload and publish shared content such as images, text, audio, and other types of content, and / or perform other functions. For example, a client device may display received data, such as content posts that are sent or streamed to the client device and originate from different client devices (or directly from different client devices) or from the server system and / or network service via the server and / or network service. In some embodiments, client devices may communicate directly with each other, for example, using peer-to-peer communication between client devices as described above. In some embodiments, a "user" may include one or more programs or virtual entities, as well as a person interfacing with the system or network.

[0040] In some embodiments, any one of client devices 120, 122, 124, and / or 126 may provide one or more applications. For example, as Figure 1 shown, client device 120 may provide image application 106b. Client devices 122 - 126 may also provide similar applications. Image application 106b may be implemented using the hardware and / or software of client device 120. In different embodiments, image application 106b may be a stand-alone client application, for example, executing on any one of client devices 120 - 124, or may work in cooperation with image application 106b set up on server system 102.

[0041] Image application 106 may provide various features implemented with user permission that are related to images. For example, such features may include one or more of the following: capturing an image using a camera, modifying an image, determining image quality (e.g., based on factors such as face size, blurriness, number of faces, image composition, lighting, exposure, etc.), storing an image or video in image library 108, providing a user interface to view an image or image-based creations, etc.

[0042] In some embodiments, with user permission, the features provided by image application 106 may include analyzing an image to determine one or more persons depicted in the image (e.g., using one or more user-permitted techniques such as face detection, face recognition, gait recognition, pose recognition, etc.), and generating a person identifier for each person (e.g., human, pet) depicted in the image in image library 108. The person identifiers for each image may be stored associated with the images in image library 108. In some embodiments, generating a person identifier may further include obtaining a thumbnail representation of the person depicted in the image, e.g., a close-up of the person, which is obtained by scaling or cropping the image depicting the user.

[0043] In some embodiments, with user permission, the Image Application 106 can analyze images to generate data and store such data in the Image Library 108. For example, the Image Application 106 can analyze images and associated metadata (e.g., location, timestamp, etc.) to group pictures into multiple segments. In some embodiments, the segments can be associated with events (e.g., birthdays or other celebrations, trips) or specific time periods. The Image Application 106 can (iteratively) analyze the images in the Image Library 108 to determine multiple segments. After determining the segments, the Image Application 106 can update the Image Library 108 to indicate the segments associated with each image in the image library.

[0044] Although the above description relates to various features of the Image Application 106, it should be understood that in various embodiments, the Image Application 106 can provide fewer or more features. Additionally, options are provided to each user for enabling and / or disabling certain features. The features of the Image Application 106 are specifically implemented upon obtaining user permission.

[0045] The Client Device 120 can include an Image Library 108b of User U1, which can be a stand-alone image library. In some embodiments, the Image Library 108b can be used in conjunction with the Image Library 108a on the Server System 102. For example, with user permission, the Image Library 108a and the Image Library 108b can be synchronized via the Network 130. In some embodiments, the Image Library 108 can include multiple images associated with User U1, such as images captured by the user (e.g., using the camera of the Client Device 120 or other devices), images shared with User U1 (e.g., corresponding images from the respective image libraries of other users U2 - U4), images downloaded by User U1 (e.g., from websites, from messaging applications, etc.), screenshots, and other images.

[0046] In some embodiments, the Image Library 108b on the Client Device 120 can include a subset of the images in the Image Library 108a on the Server System 102. Such an embodiment may be advantageous, for example, when the amount of storage space available on the Client Device 120 is limited.

[0047] In some embodiments, the Image Application 106 can enable the user to manage the image library. For example, the user can use the backup function of the Image Application 106b on the client device (e.g., any one of the Client Devices 120 - 126) to back up local images on the client device to the server device (e.g., the Server Device 104). For example, the user can manually select one or more images to be backed up, or specify backup settings for identifying the images to be backed up. Backing up the images to the server device can include, for example, collaborating with the Image Application 106a on the Server Device 104 to transfer the images to the server for storage by the server.

[0048] In various embodiments, the client device 120 and / or the server system 102 may include other applications (not shown), which may be applications that provide various types of functionality, such as, for example, calendars, address books, email, web browsers, shopping, transportation (e.g., taxis, trains, airline ticket reservations, etc.), entertainment (e.g., music players, video players, game applications, etc.), social networking (e.g., messaging or chatting, audio / video calls, sharing images / videos, etc.), and so on. In some embodiments, one or more other applications may be stand-alone applications executed on the client device 120. In some embodiments, one or more other applications may access the server system, e.g., the server system 102, which provides data and / or functionality for the other applications.

[0049] The user interfaces on the client devices 120, 122, 124, and / or 126 may enable the display of user content and other content, including images, image-based creations, data, and other content as well as communications, privacy settings, notifications, and other data. Such user interfaces may be displayed using a combination of software on the client device, software on the server device, and / or client software and server software executed on the server device 104 (e.g., application software or client software that communicates with the server system 102). The user interface may be displayed by a display device of the client device or the server device (e.g., a touch screen or other display screen, a projector, etc.). In some embodiments, an application running on the server system may communicate with the client device to receive user input data at the client device and output data at the client device, such as visual data, audio data, etc.

[0050] For ease of illustration, Figure 1 one box is shown for the server system 102, the server device 104, and the image database 110, and four boxes are shown for the client devices 120, 122, 124, and 126. The server boxes 102, 104, and 110 may represent multiple systems, server devices, and network databases, and these boxes may be provided in configurations different from those shown. For example, the server system 102 may represent multiple server systems that may communicate with other server systems via the network 130. In some embodiments, for example, the server system 102 may include cloud-hosted servers. In some examples, the image database 110 may be stored on a storage device disposed in a server system box separate from the server device 104 and may communicate with the server device 104 and other server systems via the network 130.

[0051] Moreover, any number of client devices may exist. Each client device may be any type of electronic device, e.g., a desktop computer, a laptop computer, a portable or mobile device, a cell phone, a smart phone, a tablet computer, a television, a TV set-top box or entertainment device, a wearable device (e.g., display glasses or goggles, a wrist watch, headphones, an armband, jewelry, etc.), a personal digital assistant (PDA), a media player, a gaming device, etc. Some client devices may also have a local database similar to database 106 or other storage. In some embodiments, network environment 100 may not have all of the components shown and / or may have other elements, including other types of elements in place of or in addition to those described herein.

[0052] Other embodiments of the features described herein may use any type of system and / or service. For example, instead of or in addition to a social networking service, other networking services (e.g., connected to the Internet) may be used. Any type of electronic device may utilize the features described herein. Some embodiments may provide one or more of the features described herein on one or more client or server devices that are disconnected from or intermittently connected to a computer network. In some examples, a client device including or connected to a display device may display content posts stored on a storage device local to the client device, e.g., an image previously received via a communication network.

[0053] Figure 2 is a flowchart illustrating an example method 200 for providing a user interface including image-based creation according to some embodiments. In some embodiments, method 200 may be implemented, for example, on a server system 102 as shown in Figure 1 shown. In some embodiments, some or all of method 200 may be implemented on one or more client devices 120, 122, 124, 126, shown in Figure 1 shown, on one or more server devices, and / or on both server and client devices. In the example described, the implementation system includes one or more digital processors or processing circuits (“processors”), and one or more storage devices (e.g., a database or other memory). In some embodiments, different components of one or more servers and / or clients may execute different blocks or other portions of method 200. In some examples, a first device is described as executing a block of method 200. Some embodiments may have one or more blocks of method 200 executed by one or more other devices (e.g., other client devices or server devices) that may send the results or data to the first device.

[0054] In some embodiments, method 200 or a portion of the method may be automatically initiated by a system. In some embodiments, the implementing system is a first device. For example, the method (or a portion thereof) may be performed periodically (e.g., every few hours, daily, weekly, or at other suitable intervals). In some embodiments, the method (or a portion thereof) may be performed based on one or more specific events or conditions, such as a client device initiating the image application 106 to capture a new image by the client device, uploading the new image to the server system 102, a predetermined period of time having elapsed since the last execution of method 200, and / or one or more other conditions specified in the settings read by the method occur.

[0055] Method 200 may begin at block 202. At block 202, user permission is obtained to use user data in the implementation of method 200. For example, the user data for which permission is obtained may include images stored on the client device (e.g., any one of client devices 120 - 126) and / or the server device, image metadata, user data related to the use of the image management application, user responses to image-based creations, etc. The user is provided with an option to selectively provide permission to access such user data. For example, the user may choose to provide permission to access all of the requested user data, any subset of the requested data, or deny access to the requested data. In some embodiments, one or more blocks of the methods described herein may use such user data. Block 202 is followed by block 204.

[0056] At block 204, it is determined whether the remaining steps of method 200 as described herein (blocks 210 - 224) can be implemented under the permission provided by the user, for example, by accessing the data permitted by the user. If it is determined at block 204 that the permission provided by the user at block 202 is insufficient, block 204 is followed by block 206, otherwise block 204 is followed by block 210.

[0057] At block 206, the user is requested to provide permission to access user data. For example, if the user has denied permission for one or more portions of the user data that can be used in blocks 210 - 224, a user interface may be provided to the user to provide permission for these portions of the user data. The user interface may indicate to the user how the data will be used and / or provide an indication of how providing such permission may benefit the user, such as by enabling image-based creations. Alternatively, for example, if the user denies the request to access user data or otherwise indicates that they deny permission, the method ends at block 206 such that no user data is accessed. In this case, blocks 210 - 224 are not executed.

[0058] If the permissions provided by the user are sufficient, block 204 is followed by block 210. The remaining steps of the method are performed, such as blocks 210-224, where user data is selectively accessed as permitted by the user. In some embodiments, additional prompts may be provided to the user to provide access to or modify the permissions for the user data.

[0059] In block 210, a plurality of segments are obtained. For example, segments may be obtained from a user image library stored locally on a client device and / or a server device. In some embodiments, each segment may be associated with a corresponding time period and may include a corresponding set of images.

[0060] For example, a segment may correspond to a specific event. Examples of events that may be associated with a segment include, for example, a birthday party, a wedding, or other celebration; a trip or event, such as a ski vacation, a hiking trip, a beach vacation, a sporting event or concert, a family picnic, etc. A segment may last for a relatively short time (e.g., a few hours or a day for a concert or birthday party) or a relatively long time (e.g., a multi-day ski vacation).

[0061] The set of images for a segment may include images associated with the segment taken during a certain time period. For example, images associated with a hiking vacation may include images taken during that time period, including images of people hiking, walking on a trail, eating / drinking, or participating in other activities together and / or other images (e.g., images of mountains, trees, etc.).

[0062] Each image may be associated with a corresponding person identifier. In some embodiments, the person identifier may be a unique value associated with a specific person depicted in the user image library, such as a database primary key or an equivalent value. For example, an image depicting two people walking on a trail may be associated with a corresponding person identifier for each of the two people. With user permission, unique personal identifiers may be associated with each person depicted in one or more images in the user image library.

[0063] In some embodiments, when available in the user image library, the person identifier may be associated with a person name. In some embodiments, the person identifier may be associated with a representative image or thumbnail of the person. It can be understood that for operations such as retrieving images or associating person identifiers with images, the unique identifier (data value) may be utilized. In the remainder of this document, for ease of explanation, the person identifier may be used interchangeably to refer to the unique identifier (e.g., alphanumeric or other type of data value), the person name, or the person thumbnail. In some embodiments, a person may include humans and pets (e.g., when the image library has features to distinguish between different animals). For images that do not depict a person, no person identifier may be associated, or an empty identifier may be associated. Block 210 may be followed by block 212.

[0064] Figures 3A to 3F Illustrates different stages of obtaining a group of people from segments in a user image library according to some embodiments. Additional reference is made to Figures 3A to 3F to describe the remaining blocks of method 200.

[0065] In block 212, a corresponding cluster is formed for each segment. In Figures 3A to 3D , eight clusters (and correspondingly, eight segments) are identified by the numbers 1 through 8. In some embodiments, forming a cluster may include mapping all person identifiers that appear in at least one image in the image set of the segment to the cluster. For example, as Figure 3A shows, clusters 1, 2, 3, 5, 6, and 8 each include 3 person identifiers, while clusters 4 and 7 each include 2 person identifiers. In Figures 3A to 3F , each person identifier is represented by a corresponding face.

[0066] In Figures 3A to 3C , the number next to each person identifier in each figure indicates the number of images in the image set of the segment associated with that person identifier. For example, by using person recognition techniques permitted by the user, such as face recognition, gait recognition, pose recognition, etc., when a person corresponding to the person identifier is depicted in the image, the person identifier is associated with the image.

[0067] For simplicity, the terms "person identifier X" and "person X" may be used interchangeably herein. As can be seen from Figure 3A , cluster 1 includes 20 images containing person A, 15 images containing person B, and 1 image containing person C. Cluster 2 includes 50 images of person D, 35 images of person B, and 25 images of person A. Cluster 3 includes 40 images of person E, 30 images of person A, and 10 images of person B. Cluster 4 includes 35 images of person E and 20 images of person B. Cluster 5 includes 25 images of person E, 10 images of person B, and 3 images of person F. Cluster 6 includes 10 images of person B, 5 images of person E, and 4 images of person D. Cluster 7 includes 45 images of person A and 20 images of person B. Cluster 8 includes 30 images of person A, 25 images of person B, and 2 images of person E.

[0068] In some embodiments, forming a cluster may further include: determining the number of images in the image set of the segment that include the most frequent person identifier in the segment. For example, for segment 1, the number of images is 20 (since person A is the most frequent person identifier).

[0069] In some embodiments, forming the clusters may further include: removing person identifiers associated with fewer than a threshold number of images from the clusters. In some embodiments, the threshold number of images may be determined based on the number of images including the most frequent person identifier in the segment (e.g., for segment 1, 20). For example, for segment 1, the threshold number may be determined to be 5 (20% of 20), and person identifiers associated with fewer than this threshold number (e.g., person C) may be removed from cluster 1. If removing a person identifier results in a cluster with a single person identifier, such a cluster is removed before further processing.

[0070] In different embodiments, different ways may be adopted to select the threshold number. For example, as a percentage (including 20%, 30% or other values of the number of images of the most frequent person). In some embodiments, the threshold number may be selected based on the maximum size of each cluster. For example, to remove person identifiers associated with the smallest number of images in the cluster until the total number of person identifiers in the cluster is less than or equal to the threshold size. In some embodiments, the threshold number may be selected such that each cluster includes at least a minimum number of images of each person identifier associated with the cluster, such as 3 images, 5 images, etc.

[0071] Figure 3B It is shown that person identifier C is removed from cluster 1 and person identifier F is removed from cluster 5. For cluster 1, person C is associated with only one image of the cluster, which is fewer than the threshold number of images, e.g., which may be determined to be 20×20% = 4. For cluster 5, person F is associated with 3 images of the cluster, which is fewer than the threshold number of images, e.g., which may be determined to be 25×20% = 5, where 25 is the number of images including the most frequent person identifier (person B) in cluster 5.

[0072] Selecting the threshold number of images can ensure that person identifiers relatively frequently depicted in the clusters are important persons in the segment, for example. For a segment including an image set of a group of people who went on a hiking vacation, it is likely that the frequency of depicting a part of the people in the hiking group in the images of the segment is higher than that of other people, such as other hikers, tour guides, etc. In another example, for a segment including an image set from a wedding, each person in the bride's entourage is likely to be the most important person, and these people are likely to be depicted more frequently in the image set of the segment. Thus, removing person identifiers associated with fewer than the threshold number of images ensures that the image-based creation based on the clusters includes important persons from the segment.

[0073] Block 212 may be followed by block 214.

[0074] In block 214, it is determined whether one or more person identifiers are included in fewer than a threshold number of clusters. For example, as Figure 3C shown, the person identifier D is identified as being included in two clusters (cluster 2 and cluster 6), which is fewer than the threshold number of clusters, e.g., 4. In some embodiments, the threshold number of clusters may be selected based on the total number of segments in the user image library (or the total number of segments having clusters with at least two person identifiers). For example, for a small image library (e.g., having 10 segments, 20 segments), the threshold number of clusters may be selected as a relatively low value (e.g., 2, 3, etc.), while for a large image library (e.g., having 50 segments, 200 segments or even more segments), the threshold number may be selected as a relatively high value (e.g., 5, 10, etc.).

[0075] The selection of the threshold number can be made automatically based on the library size such that using the threshold number is highly likely to result in clusters that include persons who are depicted frequently in the user image library over a significant period of time, e.g., persons who are relatively more important to the user. For example, persons who are more important to the user may include: close family members such as spouse, children, parents, siblings, etc.; friends; a subset of colleagues; etc.; while persons who are less important to the user may include other persons such as those with whom they dine, members of a tour group, persons inadvertently captured in their images, etc.

[0076] If one or more person identifiers are included in fewer than the threshold number of clusters, block 214 is followed by block 216. Otherwise, block 214 is followed by block 218.

[0077] In block 216, one or more person identifiers identified in block 214 are removed from the clusters that include these identifiers. For example, Figure 3C shows the removal of person D from clusters 2 and 6. If removing one or more person identifiers results in a cluster with a single person identifier, such a cluster is removed before further processing. Block 216 may be followed by block 218.

[0078] In block 218, the same clusters are merged to obtain groups of persons. Each group of persons includes at least two person identifiers. For example, Figure 3D shows the merging of clusters 1, 2, 7, and 8 (where each cluster includes persons A and B) to obtain group of persons 310, and the merging of clusters 4, 5, and 6 (where each cluster includes persons B and E) to obtain group of persons 330. Group of persons 320 is the same as unique cluster 3. In Figure 3D it shows the segment numbers corresponding to each group of persons next to the person identifiers of each group of persons.

[0079] In some embodiments, combining like clusters can include associating a corresponding set of segments (determined based on the combined clusters) with each of a plurality of groups of individuals. For example, as Figure 3D shown, segments 1, 2, 7, and 8 are associated with group of individuals 310, while segments 4, 5, and 6 are associated with group of individuals 330. Associating segments with groups of individuals enables selection of images from the segments when generating image-based creations for the groups of individuals, as further described below with reference to block 224. Block 218 can be followed by block 220.

[0080] In block 220, which is executed after block 218, it is determined whether at least one group of individuals is associated with fewer than a threshold number of segments. For example, as Figure 3D shown, group of individuals 320 is associated with a single segment (segment 3). In some embodiments, the threshold number of segments can be selected as two, three, or any other number. Selecting a higher threshold number of segments can enable image-based creations with greater temporal diversity, since image-based creations are based on groups of individuals and associated segments, as described below with reference to block 224. Selecting a lower threshold number of segments can enable generation of a greater number of image-based creations. For example, by selecting the threshold as two, groups of individuals associated with even two segments can be used for image-based creations, as described below with reference to block 224.

[0081] If, in block 220, it is determined that at least one group of individuals is associated with fewer than a threshold number of segments, then block 220 is followed by block 222. Otherwise, block 220 is followed by block 224.

[0082] In block 222, at least one group of individuals (identified in block 220 as being associated with fewer than a threshold number of segments) is combined with one or more other groups of individuals. In some embodiments, each of the one or more other groups of individuals combined with the at least one group of individuals includes a subset of the individual identifiers included in the at least one group of individuals. For example, Figure 3E shows group of individuals 320 (associated with a single segment) being combined with group of individuals 310 (which includes individuals A and B) and group of individuals 330 (which includes individuals B and E).

[0083] In some embodiments, combining groups of individuals can include associating the segments associated with the at least one group of individuals with the one or more other groups of individuals with which the at least one group of individuals is combined. For example, after the combination, as Figure 3F shown, segment 3 is associated with group of individuals 310 and group of individuals 330. Associating segments with groups of individuals enables selection of images from the segments when generating image-based creations for the groups of individuals, as further described below with reference to block 224.

[0084] Block 222 may be followed by block 224. In some embodiments, blocks 220 and 222 are optional. In these embodiments, block 218 is followed by block 224.

[0085] In block 224, a user interface is provided that includes an image-based creation based on a particular group of people among multiple groups of people. For example, the user interface may be provided as an actionable user interface element displayed in an image application (e.g., image application 106) such that when the user selects the actionable user interface element, the corresponding image-based creation is displayed. In some embodiments, providing the user interface may include causing instructions to be displayed to a user interface of a client device to be provided to the user (e.g., via image application 106b, a web browser, etc. on client device 120).

[0086] In some embodiments, block 224 may be executed in response to detecting that at least one of the following trigger conditions is met. Example trigger conditions may be, for example, detecting that an image matching a particular group has been recently captured since the previous execution of method 200 (or a sub-part, e.g., block 224). Another example trigger condition may be detecting that the current date (at the time of execution of block 224) matches a date associated with a particular group of people. For example, when the user library includes groups of people associated with periodic events (e.g., birthdays, anniversaries, Thanksgiving, Christmas, or other holidays, etc.), block 224 may be executed such that a user interface with an image-based creation is provided when the particular event occurs or is approaching. Another example trigger condition may be detecting an event that matches a particular group of people. For example, a planned vacation (or an ongoing or just-ended vacation) with friends or family in the group may be detected based on user-permitted data such as travel data, and block 224 may be executed accordingly.

[0087] In some embodiments, when the trigger condition is detected as a top-level one, an image for the image-based creation may be downloaded in advance to a client device (e.g., any one of client devices 120 - 126). For example, if a birthday is the trigger condition for an image-based creation based on a particular group of people, the image-based creation may be generated in advance and downloaded and cached to the client device. Alternatively or additionally, images regarding the group of people may be downloaded and cached to the client device. Such a download can ensure that the image-based creation is available (or can be generated by the client device) when the trigger condition is detected.

[0088] In some embodiments, triggering can be inhibited based on the total number of groups of people in the user image library. For example, for a user with a small image library having a relatively small number of groups of people, the frequency of providing image-based creations can be lower than that for a user with a large image library. Such an advantage can be that even in the case of determining a very small number of groups of people, image-based creations can be provided at relatively regular time intervals (e.g., monthly) or special moments (e.g., birthday parties).

[0089] In some embodiments, block 224 can be executed periodically, e.g., once a day, once a week, once a month, etc. In various embodiments, block 224 can be executed in response to any combination of these or other triggering conditions.

[0090] In some embodiments, an image-based creation can be generated partially based on the triggering conditions of block 224. For example, if block 224 is executed in response to capturing a new image, the specific group of people for the image-based creation can partially or fully match the people depicted in the image. In another example, when generating an image-based creation for a specific group of people periodically (e.g., once a month, once a year in the summer, etc.), block 224 can be executed such that the subset of images selected in block 224 can include images from a corresponding earlier time period. For example, if block 224 is executed in the summer every year, a subset of images can be selected such that images from segments associated with the previous summer are selected and images from other seasons are excluded.

[0091] In some embodiments, block 224 is executed such that a specific group of people is not repeated frequently. For example, a different specific group of people is selected each time block 224 is executed, or within a relatively short time period (e.g., a month, a quarter, etc.), similar image-based creations are not displayed in the user interface, or image-based creations for a group of people are not repeated unless there are new images available for that group of people.

[0092] In some embodiments, providing the user interface can include selecting a subset of images for an image-based creation based on a specific group of people. For example, the subset of images can be selected such that each image in the subset of images depicts a person corresponding to at least two of the two or more person identifiers included in the specific group of people. In some embodiments, the subset of images can be selected such that each image in the subset is an exact match for the group of people. In other words, in these embodiments, the subset of images can depict only the people associated with the person identifiers in the group of people and no other people.

[0093] In some embodiments, a subset of images can be selected such that each image in the subset is a match to a particular group of people, but also includes one or more additional people. In other words, in these embodiments, the subset of images can depict all of the people associated with a person identifier in a particular group of people, as well as one or more additional people (who may or may not be in another group of people). In some embodiments, a subset of images can be selected such that one or more of the additional people (who are not in the particular group of people) are not part of any other group of people.

[0094] In further embodiments, a subset of images can be selected such that each image in the subset is a partial match to a particular group of people, e.g., includes at least two people among two or more person identifiers included in the particular group of people, and optionally includes one or more additional people. In other words, in these embodiments, the subset of images can depict at least two of the people associated with a person identifier in a group of people, and optionally includes one or more additional people (who may or may not be in another group of people). In some of these embodiments, a subset of images can be selected such that one or more of the additional people (who are not in the particular group of people) are not part of any other group of people.

[0095] In some embodiments, a subset of images can be selected such that the subset includes images from at least two of a plurality of segments. For example, such embodiments can ensure that an image-based creation has temporal diversity. In other words, in such embodiments, an image-based creation depicts images from a plurality of different time periods or events in which the user library includes images. In some embodiments, e.g., when an image-based creation is to be generated with greater temporal diversity, a subset of images can be selected to include images from a greater number of segments (e.g., three segments, five segments, ten segments, etc.). For example, a certain number of images can be selected from each time period (e.g., each quarter, each year, etc.) or from each segment. In some embodiments, segments can be associated with corresponding segment salience scores, and a subset of images can be selected based on a particular number of high-quality segments (as determined based on the salience scores).

[0096] In some embodiments, a subset of images can be selected based on the current date of execution of block 224. For example, when a subset is selected for a particular group of people at regular time intervals (e.g., monthly) such that an image-based creation for the particular group of people is provided once a month, a subset of images can be selected from the images of the corresponding months of previous years, e.g., to provide an image-based creation in the form of "A and B in July" to the group of people. Other time periods (such as seasons, quarters, etc.) can also be used to select photos.

[0097] In some embodiments, a subset of the images can be selected such that the subset includes images that provide one or more of location diversity, pose diversity, or visual diversity. For example, when a user has provided access permission for locations associated with images in a user image library (e.g., as determined via image analysis or from image metadata), then a subset of the images is selected such that images from multiple locations (e.g., two, three, or more locations) are included in the subset.

[0098] In some embodiments, when a user has provided access permission for pose data associated with images in a user library, such data can be used to select a subset of the images such that the images provide pose diversity. For example, the pose data can indicate the pose of one or more persons (including person identifiers associated with a particular group of persons) in the image, such as sitting, standing, dancing, jumping, bending, squatting, high-fiving, etc.

[0099] In some embodiments, when a user has provided access permission for other image-related data associated with images in a user library, such data can be used to select a subset of the images such that the images provide visual diversity. For example, such data can indicate data about the objects depicted in the image, such as mountains, landmarks, oceans, flowers, animals, man-made objects (e.g., cakes, bicycles, yachts, etc.), etc.

[0100] In some embodiments, a subset of the images can be selected for image-based creation based on the corresponding set of images included in a segment associated with a particular group of persons. For example, a subset of the images can be selected such that each image in the subset depicts a person corresponding to at least two of two or more person identifiers included in a particular group of persons, and each image is selected from one of the segments associated with the particular group.

[0101] In some embodiments, a subset can be selected to exclude certain types of images, e.g., blurred images, images depicting a face in a small size, images not meeting quality standards, inappropriate images, images with special viewer requirements for image formats (e.g., 360-degree images, 3D images, etc.), images with extreme aspect ratios (e.g., extremely wide or long images not suitable for the client device screen), images with duplicates (or near-duplicates) in a subset (e.g., images depicting the same object or the same person with a similar pose, etc.), images that the user has manually hidden (e.g., images of a person the user no longer has a relationship with), and so on. Images used to depict a person who has been manually hidden will not be selected. The user can also select to hide for one or more time periods, such that, for example, by excluding segments that occur within or overlap with at least one of the one or more time periods, images from these time periods are excluded from the image-based creation. For example, the user can hide a specific date, date range, etc.

[0102] In some embodiments, based on the permissions provided by the user, any combination of the above factors, e.g., all or a subset of the person identifiers of the people in a group of people; including or excluding other people not in the group of people; segments from which to select a subset of images; location diversity, pose diversity, or visual diversity; segments associated with the group of people; and so on, can be used to select a subset of images for image-based creation.

[0103] In some embodiments, providing an image-based creation can further include generating an image-based creation based on a subset of images. For example, in some embodiments, the image-based creation can be a slide show of a subset of images. In these embodiments, generating an image-based creation can include arranging the images in the slide show in sequence and the corresponding display durations for each image. For example, the images can be organized chronologically based on the number of people depicted in each image, based on whether the image exactly matches a specific group of people, the visual content of the image (e.g., smiling faces, poses, objects depicted in the image), the location associated with the image, or other factors. In some embodiments, the image-based creation can include an image collage, an image album, a video (e.g., a short video), etc. In some embodiments, the user interface can provide the user with an option to share the image-based creation with other users (e.g., those in the group of people or other users).

[0104] In various embodiments, the blocks of method 200 may be combined, divided into multiple blocks, executed in parallel, or executed asynchronously. In some embodiments, one or more blocks of method 200 may not be executed. For example, in some embodiments, blocks 214 and 216 may not be executed, and block 218 may be executed after block 214. In some embodiments, block 224 may be executed multiple times, e.g., to select a subset of images for each of multiple groups of people and generate multiple image-based creations provided in the user interface.

[0105] Method 200 or some parts thereof may be repeated any number of times by using additional input. For example, in some embodiments, method 200 may be executed when one or more new segments are added to the user image library. In these embodiments, executing method 200 may include updating a previously obtained group of people, or obtaining one or more additional groups of people. In some embodiments, blocks 210 - 222 may be executed at an initial time to obtain a group of people, and block 224 may be executed at a later time, e.g., to provide the user interface on demand (e.g., when the user accesses image application 106).

[0106] Figure 4A An example user interface 400 is shown in accordance with some embodiments described herein. The user interface 400 may be displayed on a client device, such as any one of client devices 120 - 126. In some embodiments, the user interface 400 may include actionable user interface elements 402, a title 404 of the actionable user interface elements, and an image grid 406.

[0107] The image grid 406 may include multiple images of the user image library. For simplicity, Figure 4A a grid is shown with 3 images per row. In various embodiments, the grid may include any number of images per row and may be scrollable. In some embodiments, the images in the image grid 406 may be arranged in reverse chronological order.

[0108] The actionable user interface (UI) elements 402 may be associated with corresponding image-based creations. In some embodiments, when referring above to Figure 2 and Figures 3A to 3FWhen generating the actionable user interface element 402 based on a group of persons, the title 404 of the actionable user interface element may include a name associated with the person identifier of the group of persons for which the image-based creation is generated. The actionable UI element 402 may include a specific image of the image-based creation. The specific image may be selected based on various factors, such as image quality (e.g., clarity, brightness, etc.), the size and / or quality of the face depicted in the image, image recency, image resolution, whether the image includes one or more smiling faces, and if the image is a video, the length of the video, e.g., videos that are too short (e.g., less than 3 seconds) or too long (e.g., more than 30 seconds) are not selected, etc. The image may be selected such that it includes only the persons associated with the group of persons. The image included in the actionable UI element 402 may be referred to as the cover image for the image-based creation.

[0109] For example, in Figure 4B the title 404 is “A and B”, indicating that the image-based creation is based on a group of persons including person A and person B. In some embodiments, for example, when the group of persons is large, or when the names of one or more persons are not available, the title 404 may indicate a subset of names, such as “A, B, and friends”. In some embodiments, the title 404 may be generated based on a subset of the images included in the image-based creation.

[0110] The user may select the actionable user interface (UI) element 402, for example, via a tap on a touch screen, a mouse click, or other types of input. In response to selecting the actionable user interface element 402, the corresponding image-based creation may be displayed to the user, as Figure 4B shown.

[0111] Figure 4B Examples of user interfaces 412 - 416 are shown. The user interface (UI) 412 - 416 displays the image-based creation. For example, when the image-based creation is a slide show including three images associated with a specific group of persons including person A and person B, these images may be displayed in sequence. For example, the UI 412 including the first image of the image-based creation may be displayed within a first time period, followed by the UI 414 including the second image within a second time period, and then the UI 416 including the third image within a third time period. In some embodiments, each of the UI 412, UI 414, and UI 416 may include the same title (associated with Figure 4Athe same as the title 404 in [reference] or the corresponding titles of the images displayed in UIs 412 to 416. In some embodiments, no titles may be displayed in UIs 412 to 416. The images in the image-based creation may be arranged in various successive orders, such as in reverse chronological order, chronological order, by the number of people, exact match with a group of people, subsequent partial match, or any other order. As shown, the images in the three UIs depict the same two individuals A and B.

[0112] Figure 5 is a block diagram of an example device 500 that may be used to implement one or more features described herein. In one example, device 500 may be used to implement a client device, e.g., Figure 1 any one of the client devices 120 - 126 shown in [reference]. Alternatively, device 500 may implement a server device, e.g., server 104. In some embodiments, device 500 may be for implementing a client device, a server device, or both a client and a server device. Device 500 may be any suitable computer system, server, or other electronic or hardware device as described above.

[0113] One or more methods described herein may be run in a stand-alone program executable on any type of computing device, a program running on a web browser, a mobile application (“app”) running on a mobile computing device (e.g., a phone, smartphone, tablet computer, wearable device (wristwatch, armband, jewelry, headgear, virtual reality goggles or glasses, augmented reality goggles or glasses, head-mounted display, etc.), laptop computer, etc.). In one example, a client / server architecture may be used, e.g., a mobile computing device (as a client device) sends user input data to a server device and receives final output data from the server for output (e.g., for display). In another example, all computations may be performed within a mobile app (and / or other apps) on a mobile computing device. In another example, computations may be split between a mobile computing device and one or more server devices.

[0114] In some embodiments, device 500 includes a processor 502, a memory 504, and an input / output (I / O) interface 506. The processor 502 can be one or more processors and / or processing circuits for executing program code and controlling the basic operations of device 500. A "processor" includes any suitable hardware system, mechanism, or component for processing data, signals, or other information. The processor can include a general-purpose central processing unit (CPU) having one or more cores (e.g., in a single-core, dual-core, or multi-core configuration), multiple processing units (e.g., in a multiprocessor configuration), a graphics processing unit (GPU), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a complex programmable logic device (CPLD), a dedicated circuit for implementing functionality, a dedicated processor for implementing neural network model-based processing, a neural circuit, a system of processors optimized for matrix calculations (e.g., matrix multiplication), or other systems. In some embodiments, processor 502 can include one or more coprocessors for implementing neural network processing. In some embodiments, processor 502 can be a processor for processing data to produce a probabilistic output. For example, the output produced by processor 502 can be imprecise or can be accurate within a range from the expected output. Processing need not be limited to a particular geographical location or have a time limit. For example, the processor can perform its functions "in real time," "offline," in "batch mode," etc. Different (or the same) processing systems can perform portions of the processing at different times and in different locations. A computer can be any processor that communicates with a memory.

[0115] The memory 504 is typically disposed in device 500 for access by the processor 502 and can be any suitable processor-readable storage medium suitable for storing instructions executable by the processor and positioned separately from and / or integrated with the processor 502, such as random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory, etc. The memory 504 can store software for operation of the processor 502 on the server device 500, including an operating system 508, an image management application 510 (e.g., which can be the same as Figure 1 image application 106), other applications 512, and application data 514. The other applications 512 can include, for example, a data display engine, a web hosting engine, a map application, an image display engine, a notification engine, a social network engine, and so on. In some embodiments, the image application 510 can include instructions that enable the processor 502 to perform some or all of the functions described herein (e.g., Figure 2 some or all of the methods).

[0116] Other applications 512 may include, for example, a map application, an image editing application, a media display application, a communication application, a web hosting engine or application, a media sharing application, and so on. One or more of the methods disclosed herein may operate in multiple environments and platforms, for example, as a stand-alone computer program that can run on any type of computing device, as a web application with a web page, as a mobile application ("app") that runs on a mobile computing device, etc.

[0117] Any of the software in the memory 504 may alternatively be stored in any other suitable storage location or computer-readable medium. Additionally, the memory 504 (and / or other connected storage devices) may store one or more messages, one or more taxonomies, an electronic encyclopedia, a dictionary, a thesaurus, a knowledge base, message data, grammar, user preferences, and / or other instructions and data used in the features described herein. The memory 504 and any other type of storage device (disk, optical disk, magnetic tape, or other tangible medium) may be considered "storage means" or "storage devices".

[0118] The I / O interface 506 may provide functionality for enabling the server device 500 to interface with other systems and devices. The interfacing devices may be included as part of the device 500 or may be separate and communicate with the device 500. For example, network communication devices, storage devices (e.g., memory and / or database), and input / output devices may communicate via the I / O interface 506. In some embodiments, the I / O interface may be connected to interface devices such as input devices (keyboard, pointing device, touch screen, microphone, camera, scanner, sensors, etc.) and / or output devices (display device, speaker device, printer, motor, etc.).

[0119] Some examples of interfacing devices that may be connected to the I / O interface 506 may include one or more display devices 520, which may be used to display content, such as images, videos, and / or user interfaces of applications as described herein. The display device 520 may be connected to the device 500 via a local connection (e.g., a display bus) and / or via a network connection and may be any suitable display device. The display device 520 may include any suitable display device, such as an LCD, LED, or plasma display screen, a CRT, a television, a monitor, a touch screen, a 3-D display screen, or other visual display device. For example, the display device 520 may be a tablet display screen provided on a mobile device, multiple display screens provided in goggles or a head-mounted headset device, or a monitor screen for a computer device.

[0120] The I / O interface 506 can interface with other input and output devices. Some examples include one or more cameras that can capture images. Some embodiments may provide a microphone for capturing sound (e.g., as part of a captured image, voice command, etc.), an audio speaker device for outputting sound, or other input and output devices.

[0121] For ease of illustration, Figure 5 a single box is shown for each of the processor 502, the memory 504, the I / O interface 506, and the software blocks 508, 510, and 512. These boxes can represent one or more processors or processing circuits, an operating system, memory, an I / O interface, an application, and / or software modules. In other embodiments, the device 500 may not have all of the components shown and / or may have other elements including other types of elements instead of or in addition to the elements shown herein. Although some components are described as performing the blocks and operations as described in some embodiments herein, any suitable components or combinations of components in the environment 100, the device 500, similar systems, or any suitable one or more processors associated with such systems can perform the described blocks and operations.

[0122] The methods described herein can be implemented by computer program instructions or code executable on a computer. For example, the code can be implemented by one or more digital processors (e.g., a microprocessor or other processing circuit) and can be stored on a computer program product including a non-transitory computer-readable medium (e.g., a storage medium) such as a magnetic, optical, electromagnetic, or semiconductor storage medium, including semiconductor or solid-state memory, magnetic tape, removable computer disk, random access memory (RAM), read-only memory (ROM), flash memory, hard disk, optical disk, solid-state memory drive, etc. The program instructions can also be included in and provided as an electronic signal, e.g., in the form of software as a service (SaaS) delivered from a server (e.g., a distributed system and / or a cloud computing system). Alternatively, one or more methods can be implemented with hardware (e.g., logic gates, etc.) or with a combination of hardware and software. Example hardware can be a programmable processor (e.g., a field-programmable gate array (FPGA), a complex programmable logic device), a general-purpose processor, a graphics processor, an application-specific integrated circuit (ASIC), etc. One or more methods can be executed as part of or a component of an application running on a system, or as an application or software running in combination with other applications and an operating system.

[0123] Although the specification has been described with respect to specific embodiments of the specification, these specific embodiments are merely illustrative and not restrictive. The concepts illustrated in the examples can be applied to other examples and embodiments.

[0124] In addition to the above description, controls may be provided to a user that allow the user to make choices regarding whether and when the systems, programs, or features described herein may enable the collection of user information (e.g., information about a user's social network, social actions or activities, profession, user preferences, or user's current location) and whether the user has been sent content or communications from the server. Further, certain data may be processed in one or more ways before it is stored or used so that personally identifiable information is removed. For example, a user's identity may be processed so that personally identifiable information cannot be determined for the user, or a user's geographic location may be generalized (such as to a city, zip code, or state level) in cases where location information is obtained so that a user's specific location cannot be determined. Thus, the user may have control over what information is collected about the user, how that information is used, and what information is provided to the user.

[0125] Note that, as will be known to those skilled in the art, the functional blocks, operations, features, methods, devices, and systems described in this disclosure may be integrated or divided into different combinations of systems, devices, and functional blocks. Any suitable programming language and programming techniques may be used to implement the routines of specific embodiments. Different programming techniques may be employed, such as procedural or object-oriented. The routines may be executed on a single processing device or multiple processors. Although steps, operations, or calculations may be presented in a specific order, the order may be changed in different specific embodiments. In some embodiments, multiple steps or operations shown as sequential in this specification may be executed simultaneously.

Claims

1. A computer-implemented method, the method comprises: obtaining a plurality of segments, wherein each segment is associated with a corresponding time period and includes a corresponding set of images and a person identifier for each image in the corresponding set of images; forming, for each segment, a plurality of clusters including corresponding clusters, wherein each cluster includes at least two person identifiers, and wherein forming the plurality of clusters includes, for each corresponding cluster: mapping all person identifiers that appear in at least one image in the set of images of the segment to the corresponding cluster; determining, from the set of images of the segment, the number of images including the most frequent person identifier of the segment, wherein the most frequent person identifier appears in the images of the set of images of the segment more than all other identifiers of the person identifier; and removing from the corresponding cluster person identifiers that appear in fewer than a threshold number of images, the threshold number of images being determined based on the number of images including the most frequent person identifier of the segment; determining whether one or more person identifiers are included in fewer than a threshold number of clusters among the plurality of clusters; if it is determined that the one or more person identifiers are included in fewer than the threshold number of clusters, removing the one or more person identifiers from the clusters including the one or more person identifiers; after the removing, merging identical clusters to obtain a plurality of groups of persons, wherein each group of persons includes two or more person identifiers, and wherein merging the identical clusters includes: associating a corresponding set of segments with each group of persons among the plurality of groups of persons based on the clusters; and providing a user interface that includes an image-based creation based on a particular group of persons among the plurality of groups of persons.

2. The computer-implemented method according to claim 1, further comprises: after the merging, determining that at least one group of persons appears in fewer than a threshold number of segments; and in response to determining that at least one group of persons appears in fewer than the threshold number of segments, combining the at least one group of persons with one or more other groups of persons, wherein each of the one or more other groups of persons includes a subset of the person identifiers included in the at least one group of persons, and wherein providing the user interface is performed after combining the at least one group of persons with the one or more other groups of persons.

3. The computer-implemented method according to claim 1, wherein providing the user interface including the image-based creation includes: selecting a subset of images from the corresponding set of images included in the segments associated with the particular group of persons for the image-based creation, wherein each image in the subset of images depicts a person corresponding to at least two of the two or more person identifiers included in the particular group of persons.

4. The computer-implemented method according to claim 1, wherein Providing the user interface includes displaying actionable user interface elements in response to selection of at least one of the following: Detecting that an image matching the specific group of people was captured after the last display of the at least one actionable user interface element; Detecting that a current date at which the user interface is provided matches a date associated with the specific group of people; and A combination thereof.

5. The computer-implemented method according to claim 1, wherein, Providing the user interface including the image-based creation includes: Selecting a subset of images for the image-based creation based on the specific group of people, wherein each image in the subset of images depicts a person corresponding to at least two of the two or more person identifiers included in the specific group of people; and Generating the image-based creation based on the subset of images.

6. The computer-implemented method according to claim 5, wherein, Each image in the subset of images depicts a person corresponding to each of the two or more person identifiers included in the specific group of people.

7. The computer-implemented method according to claim 5, wherein, Each image in the subset of images depicts a person corresponding to each of the two or more person identifiers included in the specific group of people, and each image depicts one or more additional people not included in any other group of people.

8. The computer-implemented method according to claim 1, wherein, The plurality of segments are obtained from an image library, and further includes determining the threshold number of clusters at least in part based on the size of the image library.

9. The computer-implemented method according to claim 1, wherein, The subset of images includes images from at least two of the plurality of segments, and wherein the subset of images is selected such that the subset provides one or more selected from the following group: Location diversity, wherein images captured at multiple locations are included in the subset; Visual diversity, wherein multiple different types of objects are depicted in the images; and A combination thereof.

10. A non-transitory computer-readable medium having instructions stored thereon that, when executed by a processor, cause the processor to perform operations including the following: Obtaining a plurality of segments, wherein, Each segment is associated with a corresponding time period and includes a corresponding set of images and person identifiers for each image in the corresponding set of images; Forming a plurality of clusters including corresponding clusters for each segment, wherein each cluster includes at least two person identifiers, and wherein forming the plurality of clusters includes, for each corresponding cluster: Mapping all person identifiers that appear in at least one image in the set of images of the segment to the corresponding cluster; Determine the number of images in the image set of the segment that include the most frequent person identifier of the segment, where the most frequent person identifier appears in the images of the image set of the segment more than all other identifiers of the person identifier; and Remove person identifiers that appear in fewer than a threshold number of images from the corresponding clusters, the threshold number of images being determined based on the number of images that include the most frequent person identifier of the segment; Determine whether one or more person identifiers are included in fewer than a threshold number of the multiple clusters; If it is determined that the one or more person identifiers are included in fewer than the threshold number of clusters, remove the one or more person identifiers from the clusters that include the one or more person identifiers; After the removal, merge the same clusters to obtain multiple groups of persons, where each group of persons includes two or more person identifiers, and where merging the same clusters includes: associating a corresponding set of segments with each group of persons in the multiple groups of persons based on the clusters; and Provide a user interface that includes an image-based creation based on a particular group of persons in the multiple groups of persons.

11. The non-transitory computer-readable medium according to claim 10, wherein, the operations further include: after the merging, determine that at least one group of persons appears in fewer than a threshold number of segments; and in response to determining that at least one group of persons appears in fewer than the threshold number of segments, combine the at least one group of persons with one or more other groups of persons, where each of the one or more other groups of persons includes a subset of the person identifiers included in the at least one group of persons, and where providing the user interface is performed after combining the at least one group of persons with the one or more other groups of persons.

12. The non-transitory computer-readable medium according to claim 10, wherein, providing the user interface that includes the image-based creation includes: selecting a subset of images for the image-based creation based on the particular group of persons, where each image in the subset of images depicts a person corresponding to at least two of the two or more person identifiers included in the particular group of persons; and generating the image-based creation based on the subset of images.

13. The non-transitory computer-readable medium according to claim 10, wherein, providing the user interface that includes the image-based creation includes: selecting a subset of images for the image-based creation from the corresponding image sets included in the segments associated with the particular group of persons.

14. A computing device, comprising: a processor; and a memory coupled to the processor and having instructions stored thereon that, when executed by the processor, cause the processor to perform operations including the following: Obtain a plurality of segments, where each segment is associated with a corresponding time period and includes a corresponding set of images and a person identifier for each image in the corresponding set of images; Form a plurality of clusters for each segment, where each cluster includes at least two person identifiers, and forming the plurality of clusters includes, for each corresponding cluster: Map all person identifiers that appear in at least one image in the set of images of the segment to the corresponding cluster; Determine, from the set of images of the segment, the number of images that include the most frequent person identifier of the segment, where the most frequent person identifier appears in the images of the set of images of the segment more than all other identifiers of the person identifier; and Remove from the corresponding cluster person identifiers that appear in fewer than a threshold number of images, where the threshold number of images is determined based on the number of images that include the most frequent person identifier of the segment; Determine whether one or more person identifiers are included in fewer than a threshold number of clusters among the plurality of clusters; If it is determined that the one or more person identifiers are included in fewer than the threshold number of clusters, remove the one or more person identifiers from the clusters that include the one or more person identifiers; After the removal, merge identical clusters to obtain a plurality of groups of persons, where each group of persons includes two or more person identifiers, and merging the identical clusters includes: associating a corresponding set of segments with each group of persons among the plurality of groups of persons based on the clusters; and Provide a user interface that includes an image-based creation based on a specific group of persons among the plurality of groups of persons.

15. The computing device according to claim 14, wherein, the operations further include: After the merging, determine that at least one group of persons appears in fewer than a threshold number of segments; and In response to determining that at least one group of persons appears in fewer than the threshold number of segments, combine the at least one group of persons with one or more other groups of persons, where each of the one or more other groups of persons includes a subset of the person identifiers included in the at least one group of persons, and where providing the user interface is performed after combining the at least one group of persons with the one or more other groups of persons.

16. The computing device according to claim 14, wherein, providing the user interface that includes the image-based creation includes: Selecting a subset of images for the image-based creation based on the specific group of persons, where each image in the subset of images depicts a person corresponding to at least two of the two or more person identifiers included in the specific group of persons; and Generating the image-based creation based on the subset of images.

17. The computing device according to claim 14, wherein, The user interface providing the image-based creation includes: selecting a subset of images from the respective image sets included in the segments associated with the specific group of persons for the image-based creation.

Images