Image selection suggestions

Abstract

A method and non-transitory computer-readable medium for suggesting image selections is provided. The method includes receiving a first user input indicating a selection of one or more first images in an image library, determining one or more first image characteristics of the one or more first images, and identifying one or more second images in the image library, each of the one or more second images associated with at least one second image characteristic that matches at least one of the one or more first image characteristics. The method further includes displaying a user interface, the user interface including the one or more second images and enabling a user to select the one or more second images.

Description

[Background technology]

[0001] REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Patent Application No. 62 / 609,543 (filed December 22, 2017, and entitled “IMAGE SELECTION PROPOSAL”), the entire contents of which are hereby incorporated by reference. will be incorporated into the subsection.

[0002] background The popularity and convenience of digital camera devices have made visual content such as digital photographs and videos ubiquitous. For example, a large number of images of various types can be captured, stored, and displayed by a user device. Some devices allow a user's images or other content to be displayed in a layout that includes content items in a collection. Typically, a view of the content items is displayed on a screen, and a user can scroll through the content items so that different portions of the collection of content items are displayed in the view. A user may also select one or more content items for various purposes, such as sharing with other users, creating a picture collage or photo book, etc. Summary of the Invention [Problem to be solved by the invention]

[0003] The background art description provided herein is intended to generally present the context of the present disclosure. The inventors' currently disclosed work, to the extent described in this background art section, as well as aspects of the description that may not otherwise qualify as prior art at the time of filing, are not expressly or implicitly admitted as prior art to the present disclosure. [Means for solving the problem]

[0004] overview In one embodiment, a computer-implemented method includes receiving a first user input indicating a selection of one or more first images in an image library, determining one or more first image characteristics of the one or more first images, identifying one or more second images in the image library, each of the one or more second images associated with at least one second image characteristic that matches at least one of the one or more first image characteristics, and displaying a user interface, the user interface including the one or more second images, the user interface enabling selection of the one or more second images.

[0005] These and other embodiments may each optionally include one or more of the following features: receiving a second user input indicating a selection of at least one of the one or more second images; determining one or more second image characteristics of at least one of the one or more second images in response to receiving the second user input; identifying one or more third images in an image library, each of the one or more third images being associated with at least one third image characteristic that matches at least one of the one or more first image characteristics and at least one of the one or more second image characteristics; displaying an updated user interface, the updated user interface including the one or more third images, and the updated user interface enabling selection of the one or more third images.

[0006] These and other embodiments may each optionally include one or more of the following features: generating an image album, the image album including one or more first images and at least one of the one or more second images, receiving a second user input indicating a selection of at least one of the one or more second images, generating an image collage, the image collage including the one or more first images and at least one of the one or more second images, determining the one or more first image characteristics includes determining that the image is blurred, and identifying the one or more second images includes identifying each of the one or more second images based on the blurriness of the one or more second images.

[0007] These and other embodiments may each optionally include one or more of the following features: determining the one or more first image characteristics includes determining a location associated with the first image, identifying the one or more second images includes selecting images from an image library associated with respective locations that are within a threshold distance of the location associated with the first image, and the method further includes determining an image selection context, and identifying the one or more second images is based on the image selection context. The image selection context may be used to generate an image-based creation and / or to target the images. The image-based creation may include at least one of a picture album, a picture collage, a video, or a printed publication, or any combination thereof.

[0008] These and other embodiments may each optionally include one or more of the following features: receiving a second user input indicating a selection of at least one of the one or more second images, the selection being a copy of the first image and based on at least one of the one or more first images and the one or more second images; causing a suggested action element to be displayed in a user interface. The method may include receiving a user selection of the suggested action element, and in response to receiving the user selection, performing an action associated with the suggested action element, the action including one or more of the following: archiving at least one of the one or more first images and the one or more second images, deleting at least one of the one or more first images and the one or more second images, and / or deleting at least one of the one or more first images and the one or more second images. Perform at least one automatic highlighting.

[0009] In one embodiment, a computer-implemented method includes determining context information indicative of a target software application, identifying one or more first images in an image library based at least in part on the context information, displaying a user interface, the user interface including the one or more first images, the user interface enabling selection of the one or more first images, receiving a first user input indicating a selection of at least one image of the one or more first images, and providing the selected at least one image to the target software application in response to receiving the first user input.

[0010] These and other embodiments may each optionally include one or more of the following features: determining one or more first image characteristics of the at least one selected image and identifying one or more second images in an image library, each of the one or more second images having at least one of the one or more first image characteristics; and displaying an updated user interface; The updated user interface includes one or more second images, and the user interface enables selection of the one or more second images.

[0011] These and other embodiments may each optionally include one or more of the following features: determining the context information includes determining an application type of the target software application. The application type may include an image sharing application. Identifying the one or more first images may include selecting one or more first images from an image library that meet a quality threshold. The application type may include a financial application, and identifying the one or more first images may include selecting one or more first images from an image library that are associated with an image label that includes one or more of a receipt, a document, or a screenshot. The application type may include a messaging application, and determining the context information may further include receiving identification information of participants in a messaging conversation in the messaging application, and identifying the one or more first images may include selecting an image from an image library that depicts at least one of the participants in the messaging conversation.

[0012] These and other embodiments may each optionally include one or more of the following features: determining the context information includes receiving an application context from the target software application, and identifying the one or more first images includes determining one or more semantic concepts based on the application context and selecting the one or more first images, wherein at least one image characteristic of each of the selected images matches at least one of the semantic concepts.

[0013] In an embodiment, a non-transitory computer readable medium includes instructions stored thereon that, when executed by one or more hardware processors, cause the one or more hardware processors to perform operations including receiving a first user input indicating a selection of one or more first images in an image library; determining one or more first image characteristics of the one or more first images; identifying one or more second images in the image library, each of the one or more second images being associated with at least one second image characteristic that matches at least one of the one or more first image characteristics; displaying a user interface, the user interface including the one or more second images, the user interface enabling selection of the one or more second images; receiving a second user input indicating a selection of at least one of the one or more second images; causing the user interface to display a suggested action element based on at least one of the one or more first images and the one or more second images; and performing an action associated with the suggested action element in response to receiving a user selection of the suggested action element. In an embodiment, the action includes one or more of the following: archiving at least one of the one or more first images and the one or more second images, deleting the one or more first images and at least one of the one or more second images, generating an image-based creation that includes the one or more first images and at least one of the one or more second images, or performing automatic enhancement of the one or more first images and at least one of the one or more second images. [Brief description of the drawings]

[0014] [Figure 1] FIG. 1 is a block diagram of an example network environment that may be used for one or more embodiments described herein. [Diagram 2] 1 is a flow diagram illustrating an exemplary method for providing image selection suggestions according to one embodiment. [Diagram 3] 1 is a flow diagram illustrating an exemplary method for providing image selection suggestions according to one embodiment. [Figure 4] 5 is a schematic diagram of an exemplary user interface 500 including image selection suggestions, according to one embodiment. [Figure 5A] 4 is a schematic diagram of an exemplary user interface 400 including image selection suggestions, according to one embodiment. [Figure 5B] 5 is a schematic diagram of another exemplary user interface 530 including image selection suggestions, according to one embodiment. [Figure 5C] 5 is a schematic diagram of another exemplary user interface 540 including image creation and image selection suggestions according to one embodiment. [Figure 6] FIG. 1 is a block diagram of an example device that may be used in one or more embodiments described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] One or more embodiments assist a user in selecting an image. One or more embodiments described herein include providing suggested images for a user to select. Implementations provide benefits by providing a user interface that allows a user to select an image with reduced input, such as searching for an image using a keyword or phrase, browsing an image library by scrolling, etc. Implementations can reduce computing resources used to support a user searching or browsing by providing a user interface that allows a user to select an image without such activity.

[0016] Users select images for various purposes, e.g., from an image library, to share the images with other users, to generate image-based creations such as image albums, image collages, printed photo books, or videos that include the images. A user may have an image library that includes a large number of images, e.g., hundreds, thousands, or even millions of images. Finding and selecting images may be difficult for a user. For example, if images in an image library are organized and displayed in chronological order, a user may need to perform multiple scrolling operations to browse and select images for a particular purpose. Scrolling back and forth to find an image of interest may be tedious, and a user may find such scrolling frustrating. Furthermore, such user activity of scrolling through images requires the computing device to retrieve additional images from memory or storage and refresh the user interface to show the different images, requiring the use of computing resources of the computing device.

[0017] The image library application may include an image search feature. For example, a user may specify a text query, the image library application may return images from the library that are relevant to the search, and the user may select an image from the returned images. However, this requires the user to define the query. Furthermore, the user may need to specify multiple different queries to obtain an entire set of images to select from. Searching and selecting images via queries is tedious. Furthermore, a user's search activity may require the computing device to access an image library, for example, stored locally on the computing device or on a remote server over a network, and to retrieve the image based on the user-specified search query. It requires identifying images that match the search query and requires use of the computing resources of the computing device.

[0018] The embodiments described herein address some of the problems in finding and selecting images from an image library containing a large number of images. In one embodiment, a first user input is received indicating a selection of one or more first images in the image library. One or more first image characteristics of the one or more first images are determined. One or more second images in the image library are automatically identified and associated with at least one second image characteristic that matches at least one of the one or more first image characteristics, respectively. A user interface is displayed that includes the one or more second images and enables selection of the one or more second images.

[0019] By automatically identifying and presenting images that match user selections, the embodiments described herein eliminate the need to display images as a user scrolls through an image library or to display resulting images that satisfy a text search query specified by user input. When user input selects an image, additional images from the user's image library that have matching characteristics are automatically identified and presented for the user to select. In this manner, the techniques described herein present a user interface that allows efficient retrieval from image storage, display, and selection, and allows easy discovery and selection of images. By reducing or avoiding lengthy displays of scrolled images, and by reducing or avoiding multiple searches and displays of search results from a text search query, consumption of device resources can be reduced.

[0020] The described suggested image selection can enable faster and more efficient display of images in a user interface, and more efficient user interaction with the user interface to locate and select images. For example, the display of the suggested image selection allows a user to easily select an image, reducing or eliminating the need to manually provide text or other complex input specifying a search query to determine which image to select. Furthermore, the suggested image selection reduces the time and processing to display an image, and reduces the number of times a manually specified search is received to find an image. The suggested image selection is also useful when a client device does not have an easy, simple, or any text input capability for searching (e.g., a smart watch or other wearable device that does not include a keyboard or microphone). Such a feature enables interaction with a content display program with reduced user input and reduced time, thus reducing the consumption of device resources that would otherwise be required to receive, process, and display results in such a program.

[0021] Thus, a technical effect of one or more described embodiments is that the display of content data items in a user interface is provided with less computation time and fewer computational resources spent to obtain the results. For example, a technical effect of the described techniques and features is a reduction in the consumption of system processing resources utilized to display and select a particular image, as compared to a conventional system that does not provide one or more of the described techniques or features. For example, previous systems may use previous techniques for receiving a search query as text, in which case additional computational resources (with corresponding power consumption), such as processor, memory, and display resources, are required to display iterations of search results as the user determines which search queries are relevant to provide images to select.

[0022] Further technical effects of the described techniques and features include the and reducing the consumption of system processing resources, such as display and search processing, and power consumption, utilized by systems that do not provide one or more of the above. For example, in such conventional systems, a user manually scrolls through a view of content elements, the user must manually locate a desired content item through such manual scrolling, (e.g., repeatedly displaying and scrolling content data items forward and backward, displaying and / or searching for content items from the user, etc.). This can result in inefficient use of system resources (due to repeated receipt of commands for

[0023] In situations where certain embodiments described herein may collect or use personal information about a user (e.g., user data, information about the user's social network, the user's location and time at the location, the user's biometric information, the user's activity and demographic information), the user is provided with one or more opportunities to control whether information is collected, whether the personal information is stored, whether the personal information is used, and how the information is collected, stored, and used about the user. That is, the systems and methods described herein are particularly capable of collecting, storing, and / or using user personal information with explicit permission from the appropriate user to do so. For example, a user is provided with control over whether the program or feature collects user information about that particular user or other users associated with that program or feature. Each user from whom personal information is to be collected is presented with one or more options to allow control of the collection of information related to that user and to provide permission or authorization regarding whether information is collected and what portions of the information are to be collected. For example, a user may be provided with one or more such control options over a communications network. Additionally, certain data may be handled in one or more ways before being stored or used such that personally identifiable information is removed. As one example, a user's identifying information may be handled such that personally identifiable information is not determined. As another example, the geographic location of a user device may be generalized to a larger area such that the user's specific location cannot be determined.

[0024] An image as referred to herein is a digital image having pixels with one or more pixel values ​​(e.g., color values, brightness values, etc.). An image may be a still image or a single image, or may be a series of images, e.g., an image included in a frame in a video sequence of video frames, or an image in an animation of a sequence or images of different types. A video includes a sequence of multiple images. For example, the embodiments described herein may be used with content data items that are single images or still images (e.g., photographs, emojis, or other images), videos, or animated images (e.g., cinemagraphs or other animated images including motion, stickers including animation and sound, etc.). Text, as referred to herein, may include alphanumeric characters, emojis, symbols, or other characters. Audio segments may include audio data provided in a standard audio format that may be processed to provide audio, for example, from a speaker.

[0025] 1 illustrates a block diagram of an exemplary network environment 100 that may be used in certain embodiments described herein. In certain embodiments, the network environment 100 includes one or more server systems, e.g., a server system 102 and a second server system 140 in the example of FIG. 1. The server systems 102 and 140 may be in communication with a network 130, for example. The server system 102 may include a server device 104 and a database 106 or other storage device. In certain embodiments, the server device 104 may provide an image library application 156b. The second server system 140 may host one or more applications, e.g. a second server device 142 configured to provide a messaging application A 144, an image sharing application B 146, and a financial application C 148; It can include.

[0026] The network environment 100 also communicates with each other and / or The second server system 140 may be connected to the second server system 102. The network 130 may include one or more client devices, e.g., client devices 120, 122, 124, and 126, that communicate with the network 130. The network 130 may be any type of communication network, including one or more of the Internet, a local area network (LAN), a wireless network, a switched or hub connection, etc. In one embodiment, the network 130 may include a network that communicates with the network 130, e.g., a peer-to-peer wireless protocol (e.g., Bluetooth, Wi-Fi Direct, etc.), etc. An example of peer-to-peer communication between two client devices 120 and 122 is shown by arrow 132.

[0027] For ease of illustration, FIG. 1 shows one block of server system 102, server device 104, database 106, second server system 140, and second server device 142, and four blocks of client devices 120, 122, 124, and 126. Server blocks 102, 104, 106, 140, and 142 may represent multiple systems, server devices, and network databases, and the blocks may be provided in configurations different than those shown. For example, server system 102 and / or second server system 140 may communicate with other servers via network 130. A server system may represent multiple server systems with which the server system can communicate.

[0028] In one embodiment, the server system 102 and / or the second server system 14 0 may include, for example, a cloud hosting server. In some examples, the database 106 and / or other storage devices may be hosted on a server separate from the server device 104. The client devices 104 and 106 may be provided in a system block and may communicate with the server device 104 and other server systems via a network 130. There may also be any number of client devices. Each client device may be any type of electronic device, such as a desktop computer, a laptop computer, a portable or mobile device, a mobile phone, a smart phone, a tablet computer, a television, a television set-top box or entertainment device, a wearable device (e.g., display glasses or goggles, a watch, a headset, an armband, jewelry, etc.), a personal digital assistant (PDA), a media player, a gaming device, etc. Some client devices may The device may also have a local database similar to database 106 or other storage. In some embodiments, network environment 100 may not have all of the components shown and / or may include components in place of or in lieu of the components described herein. In addition, it may have other components, including other types of components.

[0029] In various embodiments, end users U1, U2, U3, and U4 can communicate with the server system 102 and / or with each other using respective client devices 120, 122, 124, and 126. In some examples, users U1, U2, U3, and U4 are the respective client devices and / or server systems. and / or via an application running on the server system 102 or the second server system 140. For example, each of the client devices 120, 122, 124, and 126 may communicate with one or more server systems (e.g., the system 102, the second server system 140) through a network service provided by the client device 120, 122, 124, and 126, such as a social network service or other type of network service. Data can be communicated.

[0030] In one embodiment, the server system 102 and / or the second server system 14 0 indicates that each client device is connected to the server system 102 or the second server system 140 and / or the network service for communication content or sharing. Appropriate data can be provided to the client devices so that the content can be received. In some examples, users U1-U4 can interact via audio or video conferencing, audio, video, or text chat, or other communication modes or applications. The network services implemented by server system 102 or second server system 140 may include systems that enable users to conduct various communications, form links and associations, upload and post shared content, such as images, text, video, audio, and other types of content, and / or perform other functions. For example, a client device may receive or stream content from a server and / or network service (or from a different client). directly from a client device), content posts originating from different client devices or originating from server systems and / or network services, etc. The received data may be displayed. In one embodiment, the client devices may communicate directly with each other, e.g., using peer-to-peer communication between the client devices as described above. In one embodiment, a "user" may include one or more programs or virtual entities as well as a person that interfaces with a system or network.

[0031] In one embodiment, client devices 120, 122, 124, and / or Any of the client devices 126 may host one or more applications. For example, as shown in FIG. 1, the client device 120 may host a camera application 152, a photo library application 156a, and one or more other applications. Client devices 122-126 may also provide similar applications. For example, camera application 152 may provide a user of each client device (e.g., users U1-U4) with the ability to capture images, for example, using a camera on each client device. For example, camera application 152 may be a software application executing on client device 120. In one embodiment, camera application 152 may provide a user interface.

[0032] In one embodiment, the client device 120 may include an image library application 156a, which may be implemented as part of the hardware and / or software of the client device 120, as described with reference to FIG. In various embodiments, the image library application 156a may be implemented using, for example, a standalone application running on any of the client devices 120-124. It may be a stand-alone image library application or may work in conjunction with an image library application 156 b provided on the server system 102 . The image library application 156a and the image library application 156b can provide image management functions. The image management functions can include the ability to store photos and videos captured using the camera application 152, the ability to store other images, e.g., scanned images, screenshots, etc., image editing functions, and the ability to generate image-based creative works, such as image collages, videos, image albums, or printed publications (e.g., photo books). The image library application 156 can also programmatically analyze the images, e.g., image pixels and image metadata, to determine one or more image characteristics. The image library application 156 may be used to store, for example, local storage of the client device and / or a network Remote storage of one or more other devices communicating via network 130, such as databases 106 of server systems 102 and / or 140, one or more The image data and characteristics may be stored in the storage, etc., of a number of different client devices.

[0033] In one embodiment, client device 120 may include one or more other applications 154. For example, other applications 154 may be applications that provide various types of functionality, such as a calendar, an address book, email, a web browser, shopping, transportation (e.g., taxi, train, airline reservations, etc.), entertainment (e.g., music player, video player, gaming applications, etc.), social networking (e.g., messaging or chat, audio / video calling, image / video sharing, etc.), etc. In one embodiment, one or more of other applications 154 may be standalone applications executing on client device 120. In one embodiment, one or more of ... an email address, a web browser, a web browser, a web browser, a web browser, a web browser, a web browser, a web browser, a web browser, a web browser, a web browser, a web browser, a web browser, a web browser, a web browser, a web browser, a web browser, a web browser, a web browser, a web browser, a web browser, a web browser, a web browser, a web browser, a web browser, a web browser, a web browser, a web browser, a web browser, a web browser, a web browser, a web browser, a web browser, a web browser, a web browser, a web browser, a web browser, a web browser, a web browser, a web browser, a web browser, a web browser, a web browser, a web browser, a web browser, a web browser, a web browser, a web browser, a web browser, a A server system providing functionality, such as the second server system 140, may be accessed. For example, any of the applications 144, 146 and 148 shown as being provided by the second server system 140 may access data and / or or provide commands to one or more of the other applications 154. In one embodiment, the server applications 144-148 may provide, for example, It may be a stand-alone application accessed by a client device via a web browser or other client-side program.

[0034] User input on client device 120, 122, 124, and / or 126 The user interfaces may enable the display of user content and other content, including images, videos, data, and other content, as well as communications, privacy settings, notifications, and other data. Such user interfaces may be implemented by software on a client device, software on a server device, and / or a client device. The client software and the server device 104 and / or the second server device 14 2 in combination with server software, e.g., application software, or other software that communicates with the server system 102 and / or the second server device 142. The user interface may be displayed using client software that performs the operations on the client device. The user interface may be displayed by a display device of the client device or the server device, e.g., a touch screen or other display screen, a projector, etc. In one embodiment, an application program executing on the server system may communicate with a client device to receive user input at the client device and to output data at the client device, such as visual data, audio data, etc.

[0035] In one embodiment, the server system 102, the second server system 140, and / or Either one or more of the client devices 120-126 may provide a communications application program. The communications program may enable a system (e.g., a client device or a server system) to provide options for communicating with other devices. The communications program may provide one or more associated user interfaces that are displayed on a display device associated with the server system or the client device. The user interfaces may provide the user with various options for selecting a communication mode, a user or device to communicate with, etc. The communications program may, for example, display or otherwise display sent and received content posts in any of a variety of formats. It is possible to output

[0036] Other embodiments of the features described herein may be implemented in any type of system and / or The services can be used. For example, other network services (e.g., connected to the Internet) can be used instead of or in addition to social networking services. Any type of electronic device can utilize the features described herein. An embodiment can provide one or more features described herein on one or more client devices or server devices that are disconnected or intermittently connected to a computer network. In some examples, a client device that includes or is connected to a display device can display images stored in a storage device local to the client device, e.g., previously received over a communication network.

[0037] 2 is a flow diagram illustrating an exemplary method for providing image selection suggestions according to one embodiment. In one embodiment, method 200 may be implemented, for example, on server system 102 as shown in FIG. 1. In one embodiment, some or all of method 200 may be implemented on one or more client devices 120, 122, 124, or 126, on one or more server devices, and / or on a server system as shown in FIG. In the illustrated example, the implementation system includes one or more digital processors or processing circuits ("processors") and one or more storage devices (e.g., database 106 or other storage devices). In one embodiment, the implementation system may be implemented on different servers and / or clients. The components may perform different blocks or other portions of method 200. In some examples, a first device is described as performing blocks of method 200. An embodiment may have one or more blocks of method 200 performed by one or more other devices (e.g., other client devices or server devices) that may send results or data to the first device.

[0038] In some embodiments, method 200 or portions thereof may be initiated automatically by a system. In some embodiments, the implementing system is the first device. For example, the method (or portions thereof) may be executed periodically or may be executed in response to one or more specific events or conditions, such as an application started by the user, an image selected by the user, and / or a setting read by the method. The execution may be based on one or more other conditions that may be specified by the user in the user's stored custom preferences.

[0039] In one example, the client device may be a camera, a mobile phone, a smart phone, a tablet computer, a wearable device, or other client device capable of receiving content input (e.g., image capture) and receiving user input by a user to the client device (e.g., using a touch screen, using gestures, using a mouse or other pointing device, using a keyboard, etc.) and performing method 200. In another example, a client device or a server device may perform method 200. An embodiment may initiate method 200 based on user input. A user (e.g., an operator or end user) may, for example, select to initiate method 200 from a user interface, such as an application user interface or other user interface. In one embodiment, method 200 may be implemented by a client device. In one embodiment, method 200 may be implemented by a server device. This can be implemented.

[0040] An image, as referred to herein, may include a digital image having pixels with one or more pixel values ​​(e.g., color values, brightness values, etc.). Images may include still images (e.g., still photographs, images having a single frame, etc.), moving images (e.g., animations, animated GIFs, cinemagraphs where some parts of the image include movement and other parts are still, etc.), and The text may be still images, video, or video (e.g., a sequence of images or image frames that may include audio). Although the remainder of this specification refers to images as still images, it should be understood that the techniques described herein are applicable to moving images, videos, and the like. For example, the embodiments described herein may be used with still images (e.g., photographs, emojis, or other images), videos, or moving images). Text, as referred to herein, may include alphanumeric characters, emojis, symbols, or other characters.

[0041] At block 202, in the implementation of method 200, it is checked whether user consent (e.g., user permission) has been obtained to use user data. For example, user data may include images captured by a user using a client device, e.g., images stored or accessed by a user using a client device, image metadata, user data related to use of image sharing applications, user data related to use of messaging applications, user data related to use of financial applications, user data related to use of social networking applications, user preferences, user biometric information, user characteristics (such as identity, name, age, gender, occupation, etc.), information about the user's social network and contacts, social and other types of actions and activities, content created or submitted by the user, ratings, and opinions, the user's current location, past user data, images generated, received, and / or accessed by the user, content viewed by the user, and the like. User data may include user views, shared images, etc. One or more blocks of the methods described herein may use such user data in some embodiments.

[0042] If user consent has been obtained from an associated user whose user data may be used in method 200, then at block 204 it is determined that the blocks of the method herein may be performed with possible use of the user data as described with respect to those blocks, and the method continues to block 212. If user consent has not been obtained, then at block 206 it is determined that the blocks should be performed without use of user data, and the method continues to block 212. In one embodiment, if user consent has not been obtained, then the blocks are performed without use of user data, using synthetic data and / or generic or publicly accessible data. The method is performed using available and publicly available data. If user consent has not been obtained, method 200 is not performed.

[0043] At block 212 of method 200, a selection of a first image is received. For example, a number of images from a user's image library may be displayed on a display screen of a device. In an embodiment, image thumbnails may be displayed and selection of a thumbnail may correspond to selection of a corresponding image. For example, the images or thumbnails may be displayed in a grid configuration, e.g., three images per row, four images per row, etc.

[0044] In one embodiment, the images may be grouped by one or more image characteristics, for example, by a date or time associated with the corresponding images, by a location (e.g., a geographic location) associated with the corresponding images, by the image album to which the corresponding images belong, etc. In one embodiment, a user may use image tagging to group images using facial recognition and / or by people depicted in the images. If you grant permission for use of the image, the image may be, for example, "Andrew and Enika" or "Andrew In one embodiment, the groups can utilize multiple image characteristics or factors, e.g., "Andrew and Enika on a forest path," "Yesterday with Joyce," etc.

[0045] In one embodiment, a first user input is received indicating a selection of one or more first images. In one embodiment, receiving the user input may include detecting a tap or press gesture on an image displayed on a touch screen, detecting voice input, gaze input, manipulation of an input device (mouse, joystick, trackpad, etc.), etc. The user may select one or more of the multiple images by selecting a corresponding image element displayed on the screen (e.g., an image thumbnail or other image representation, or a display of the entire image). In one embodiment, the images may be scrollable, or the user may scroll through various and / or multiple images of the multiple images. The images may be organized into multiple pages or sections so that additional images can be selected. Upon receiving a user selection of one or more first images, the method proceeds to block 214.

[0046] At block 214, first image characteristics of the selected images, e.g., one or more first images, are determined. In an embodiment, determining the first image characteristics may include retrieving the image characteristics from image metadata stored in association with the first images. In an embodiment, determining the first image characteristics may include retrieving the image characteristics from an image database, e.g., an image library that stores images and image characteristics in a database.

[0047] In one embodiment, the first image characteristics include image content (e.g., pixel values ​​of the first image) and / or image metadata (e.g., date of capture, location of capture, camera, etc.). Based on the analysis of the camera make / model, camera settings, image resolution, image color depth, etc. In some embodiments, the analysis of image content may be performed offline, e.g., prior to receiving a user's selection of an image. For example, the analysis of the image content of an image may be performed when the image is generated (e.g., captured using a camera, captured as a screenshot, etc.) or when it is added to a library (e.g., downloaded or stored on a device having an image library).

[0048] In one embodiment, the first image characteristic may be determined by programmatically analyzing the image. In one embodiment, the first image characteristic may include one or more labels associated with the image. In one embodiment, the one or more labels may be based on image pixel values ​​and / or image metadata. In one embodiment, the one or The multiple labels may include one or more semantic concepts associated with the image. In an embodiment, the semantic concepts may be organized into a hierarchy of concepts, e.g., "food->cake->birthday cake", "sports->baseball->equipment->bat", and the narrower concepts may be included in the broader concept categories. There may be any number of semantic concepts associated with the image. The semantic concepts may be determined based on one or more of detecting one or more objects in the image, image attributes such as focus, lighting levels, image composition, color distribution, metadata such as location, time, and other metadata, and other factors. In an embodiment, the semantic concepts may be determined using a machine learning model trained to recognize semantic concepts from images.

[0049] In one embodiment, image characteristics, e.g., one or more labels, may be associated with a confidence score. The confidence score may indicate a degree of confidence that a particular label is applied to an image. The one or more labels and corresponding confidence scores may be based on programmatically analyzing the image. In one embodiment, programmatically analyzing the image may may be performed using a trained machine learning model. In some embodiments, programmatically analyzing the image may be performed using object detection and recognition techniques. In some embodiments, a combination of various image analysis techniques may be used.

[0050] In one embodiment where a user provides consent, programmatically analyzing the image may include identifying one or more semantic concepts from the image pixel data. For example, it may be determined whether the image was taken indoors or outdoors. In another example, the machine learning model may determine whether the image shows one or more of various objects, such as a tree, a flower, a building, a river or ocean, a mountain, etc. In one embodiment, the machine learning model may determine specific characteristics of the objects shown, such as, for example, a "red rose," a "12-story building," a "coffee shop," a "birthday cake," etc.

[0051] In some embodiments where the user provides consent, one or more objects in the image may be recognized, e.g., "Taj Mahal," "baseball bat," "ice cream," etc. In embodiments where the user provides consent to the use of face detection, it may be determined whether the image shows one or more people (e.g., "image has three people" or "image shows two women"). In embodiments where the user provides consent to the use of face recognition technology, people in the image may be recognized, e.g., "Joyce," "Andrew," etc.

[0052] In one embodiment where a user provides consent, image characteristics may be determined from image metadata. For example, such image characteristics may include camera make / model, camera settings (e.g., aperture, , flash usage, camera mode, etc.), and the date and / or time of image capture , which may include the location of image capture (if the user allows the use of location data).

[0053] In one embodiment in which a user provides consent, one or more semantic concepts may be determined based on objects represented in the image and / or image metadata. For example, when an image is captured with the flash on, the objects shown include a dining table and food, and the location and time of image capture indicate that the image was captured at night, the semantic concepts may be "dinner" and "indoors." In another example, when an image shows water and a boat, the semantic concept of "sailing" may be associated with the image.

[0054] Recognizing objects in an image and / or determining the semantic concepts that correspond to the image At times, one or more labels can be associated with an image. Image characteristics, such as the label or labels, can be stored in a database, for example, as image metadata in an image file. For example, associated labels can be "Joyce, dinner, indoors, house," "Hiking, Yosemite National Park," etc.

[0055] In one embodiment, the image characteristics may include an image type, e.g., “document,” “receipt,” “screenshot,” “photo,” etc., determined based on programmatically analyzing the image. In one embodiment, the image characteristics may include image attributes, e.g., “blurry,” “dark,” etc., determined based on programmatically analyzing the image. Block 214 may be followed by block 216.

[0056] At block 216, one or more second images are identified in the image library. In one embodiment, the one or more second images may have at least one image characteristic that is the same as or similar to one or more first image characteristics of the one or more first images selected by the user. For example, if the user selects a first image that is blurry, one or more second images each associated with the image characteristic of blurry may be selected. The image is identified.

[0057] In another example, if a user selects two images of a document (e.g., images showing text or visual content determined to be a document), one or more second images are identified, each associated with the image property "document." In another example, if a user selects two images that are screenshots (e.g., captures of the contents of a device's display screen), one or more second images that are each screenshot are identified and associated, e.g., with the image property "screenshot." In another example, if a user selects images showing two individuals, e.g., Joyce and Andrew, one or more second images are identified, each associated with at least one image label of "Joyce" and "Andrew."

[0058] In another example, if one or more of the first images are associated with a location, e.g., "Taj Mahal," one or more second images associated with the same or similar location, e.g., "Taj Mahal," "Agra," etc., may be selected. In an embodiment, the one or more second images may be selected such that the location associated with the second image is within a threshold distance of the location associated with the first image. For example, the threshold distance may be a numerical distance, e.g., "5 miles." In an embodiment, the one or more second images may be selected based on matching of locations, e.g., city names, country names, etc. In an embodiment, the one or more second images may be selected based on images that exhibit one or more of the same characteristics as depicted by the first image at the first location, e.g., monuments, bridges, or skyscrapers, etc.

[0059] In an embodiment, a context of the image selection may be determined. For example, the user interface in which the user provides the first user input may be determined to be associated with generating an image-based creation with one or more devices, such as, for example, an image collage, a video based on images in an image library, an image album, or a printed publication, such as a photo book. In an embodiment, the context of the image selection may be or may be based on an associated target software application in which the user interface for image selection was initiated, such as, for example, a messaging application, an image sharing or social networking application, a financial application, etc.

[0060] In embodiments in which a context for image selection is determined, identifying the one or more second images may be based on the context. For example, the context may be associated (e.g., prior to block 216) with one or more selection criteria for selecting the second image. For example, if the context is determined as generating or sharing a photo book via an image sharing application, the one or more second images may be selected based on quality criteria associated with the context, e.g., image resolution criteria, image color depth criteria, image focus criteria, etc. In some examples, images that do not meet the image resolution threshold criteria may be excluded from the one or more second images, even if such images have characteristics that match the first image characteristics. In another example, if the context is determined as a financial application, images having certain characteristics, e.g., image types "receipt," "bill," "financial statement," "document," etc., may be included in the one or more second images, and other images may be excluded.

[0061] In one embodiment, the one or more second images are selected or filtered based on some image characteristics that match the image characteristics of the first image. For example, if the first image shows "Joyce and Andrew at the beach", the second image is selected or filtered based on how many of the three characteristics "Joyce", "Andrew" and "beach" match each of the second images. In one example, if an image library includes image A, image B, and image C having all three characteristics, and additional images D, E, and F having two of the three characteristics, images A, B, and C may be selected as the second image or images, and images D, E, and F may be excluded from the second images. In one embodiment, images A-F are included in the second images, and other images, including images having only one of the three characteristics, are excluded from the second images.

[0062] In one embodiment, a confidence threshold can be used to filter one or more second images. For example, if a first image has the property "recipe" with a confidence score of 90%, the second image or images can be filtered. A feature "receipt" that has a confidence score of at least 90% when associated with "receipt" In this example, images with lower confidence scores may be excluded from the one or more second images, even if the images are associated with the property "receipt". In an embodiment, a combination of the number of properties that match the image properties of the first image and the confidence scores associated with the properties may be used to select the one or more second images. In an embodiment, each image property may be weighted to determine the one or more second images. For example, some properties, such as labels such as "Taj Mahal" or "Joyce", may be assigned a higher weight than other properties, such as "outdoors".

[0063] In one embodiment, when a user provides user data related to image selection and consent for use of an image-based creative work generated by the user, such data may be used to select one or more second images. If the user has previously generated a printed publication, such as, for example, a photo book, that includes photographs of flowers, one or more second images showing flowers may be included. In this example, images included in the previous photo book may be excluded, for example, if the user data indicates that the user regularly generates photo books that include recently captured images.

[0064] In another example, if the user has previously ignored a suggestion to select a certain type of photograph, e.g., a photograph that does not depict a human face, such images may be excluded from the one or more second images. In another example, if the user data indicates that the user is generating a photo collage that includes images of a particular individual, e.g., family, pets, etc., images showing that individual may be selected in the one or more second images, and images that do not depict that individual may be excluded. In an embodiment, images with certain characteristics, e.g., high blur, dark images, low quality images, archival images, etc., may be excluded from the one or more second images, even if such images have characteristics that match the one or more first images.

[0065] In an embodiment, for example, when a user indicates that the image-based creation is a video, the one or more second images may include video clips or images with motion that may be excluded for other image-based creations, e.g., an image collage. In an embodiment, the video clips may be selected based on non-visual parameters, e.g., the length of the video clip, semantic concepts or labels for determined audio portions of the video clip, etc. Block 216 may be followed by block 218.

[0066] At block 218, a user interface is caused to be displayed. For example, the user interface may be displayed by any of the client devices 120-124. The user interface may include one or more second images. For example, the user interface may be displayed as, for example, a card or overlay including a display of a grid of images. In an embodiment, the one or more second images may be displayed along with an indication that these images are suggested selections, e.g., in the user interface. The currently selected image may be displayed in a "Suggestions" section of the user interface. The one or more second images are user selectable. In an embodiment, the user interface may also include other images that are user selectable, for example, recent images captured or received within a particular time immediately prior to the current time and not included in the one or more second images. In an embodiment, the other images may be grouped in a section of the user interface separate from the "Suggestions" section, for example, a "Recent" section. In an embodiment, the currently selected image may be displayed in a section of the user interface, for example, a "Choice Tray" section. The "Choice Tray" may be separate from other portions of the user interface that include the first image and one or more additional images available for selection by the user. Block 218 may be followed by block 220.

[0067] At block 220, it is determined whether a further image selection has been received. For example, the further image selection may be received via a second user input. In an embodiment, the image indicated by the further image selection may include at least one of the one or more second images. In an embodiment, the image indicated by the further image selection may include at least one of the one or more recent images. If a further image selection has been received, the method proceeds to block 214. In an embodiment, one or more third images in an image library may be identified. For example, the one or more third images may be identified from the further image selection, e.g., from the one or more second images and / or from a user input. The selected image may be identified from other images displayed in the user interface based on determining characteristics of the selected image.

[0068] The one or more third images may be identified by matching characteristics of the first image (selected in the first user input) with the image selected via the second user input. For example, the one or more third images may be identified such that each of the third images has at least one characteristic that matches characteristics of the selected image, e.g., at least one characteristic associated with one or more of the first images and associated with one or more of the second images. For example, if the first image (selected by the first user input) shows "Joyce and Andrew on the beach" (e.g., associated with the labels "Joyce", "Andrew", "Beach") and the selected image (selected by the second user input) shows "Joyce and Andrew", then the one or more third images may be identified as images in the image library that show both Joyce and Andrew. In this example, the "Beach" characteristic is not considered while identifying the third image.

[0069] In different embodiments, the one or more third images may be selected using techniques similar to those used to select the one or more second images. In embodiments in which one or more third images are identified, the user interface may be updated to include the one or more third images. For example, updating the user interface may include updating a "Suggestions" section to display the one or more third images instead of or in addition to the one or more second images. In an embodiment, upon identifying the one or more third images, the "Suggestions" section may be displayed with a higher priority than other sections of the user interface (e.g., "Recent").

[0070] In one embodiment, further user input indicating an additional selection of an image may be received (e.g., by repeating blocks 214-220), and in response, the user interface may be updated to include the additional image based on matching image characteristics of images in the image library with image characteristics of the image selected by the user input. If no further image selection is received, block 220 may be followed by block 222.

[0071] At block 222, a user selection of one or more actions (e.g., operations) is received, e.g., via a user interface. The user selected action is performed using the images selected by the user. For example, the user may indicate that an image-based creation (e.g., a content creation) is to be generated based on the selected images, e.g., image selections indicated by the first user input, the second user input, and subsequent user inputs, if any. For example, the image-based creation may include an image collage, a video showing the selected images, e.g., as a short video clip or slide show, an image album including the selected images, or a printed publication, e.g., a photo book, including the selected images.

[0072] In another example, the one or more actions selected by the user may be providing the selected image to a target software application, such as a messaging application, an image sharing application, a social networking application, a financial application, etc. In this example, the selected image is provided to the target software application, for example, via an application programming interface (API). In an embodiment, the selected image may be modified (e.g., automatically enhanced, cropped, reduced file size, etc.) before providing the image to the target software application.

[0073] In an embodiment, the one or more actions selected by the user may be actions for an image library application, for example, to perform modifications to images in the image library. For example, if the user selects images that contain duplicates of each other or other images (e.g., selects images that all depict similar subject matter and are associated with similar metadata as one or more other selected images, or depict similar subject matter to one or more other images in the library), the action may be to remove the duplicates. For example, duplicate images that have image characteristics that score lower than other duplicate images may be removed based on visual characteristics, such as, for example, blurriness, exposure, color noise, object location relative to image boundaries, etc. In another example, if the user selects images that each have one or more common characteristics (e.g., being "dark", "unenhanced"), the action may be to perform automatic enhancement of the selected images, for example, by modifying pixel values ​​using image enhancement techniques, such as, for example, by applying image filters, by modifying the images using a machine learning model trained for image enhancement, etc.

[0074] In an embodiment, the action may be to archive the selected image, which hides the selected image from one or more views of the images in the image library, e.g., the main view. For example, a label associated with the selected image may be updated to indicate that the selected image has been archived. The archived image may be displayed when a command to view the archived images is received, e.g., via user input. In an embodiment, image metadata or labels associated with the archived image may be updated to indicate that the image is archived. Such metadata or labels may be used, for example, when generating a main view of the images in the image library, to exclude the archived image from the main view.

[0075] In some embodiments, the one or more actions may include actions corresponding to suggested action elements displayed in the user interface. In these embodiments, the suggested action elements may be determined based on the selected images, e.g., one or more of the first images and one or more of the second images. In some embodiments, the action associated with the suggested action element may be determined based on the selected images, e.g., one or more of the first images and one or more of the second images. This may include one or more of archiving the image, deleting the selected image (e.g., removing the selected image from an image library, deleting the image from storage, etc.), or performing automatic enhancement of the selected image.

[0076] For example, the selected image may be determined to have an image characteristic of "blurry." Based on the determination, a suggested action element, e.g., "remove all blur," is displayed. If the user selects the suggested action element, the corresponding action may be performed, for example, by the device. In an embodiment, additional images that match the selected image, which may include, for example, other blurry images in a library that were taken at or near the same location and at a similar time as the selected image, are performed on the user-selected image and the additional images that match the selected image.

[0077] Although method 200 has been described with reference to various blocks in FIG. 2, it should be understood that the techniques described in this disclosure may be performed without executing some of the blocks in FIG. 2. In an embodiment, one or more of the blocks shown in FIG. 2 may be combined. For example, two or more of blocks 214, 216, and 218 may be combined. In various embodiments, some of the blocks of method 200 may be performed in parallel or in a different order than that shown in FIG. 2. In an embodiment, method 200 is performed by a client device, e.g., one or more of client devices 120-124. In an embodiment, method 200 is performed by a service device, e.g., server device 104. In an embodiment, method 200 is performed by a combination of client devices and server devices. For example, in an embodiment, blocks 202, 204, 206, 212, 218, and 220 are performed by a client device, and blocks 214, 216, and 222 are performed by a server device. For example, such an approach may be useful when the client device 120 has limited ability to determine image characteristics, e.g., by programmatically analyzing the image, has limited storage capacity for storing image data, etc.

[0078] 3 is a flow diagram illustrating an example of a method 300 for providing image selection suggestions, according to one embodiment. In one embodiment, method 300 may be implemented, for example, on server system 102, as shown in FIG. 1. In one embodiment, some or all of method 300 may be implemented on one or more client devices 120, 122, 124, or 126, on one or more server devices, and / or on a server system, as shown in FIG. The implementation may be implemented on both server and client devices. In the illustrated example, the implementation system includes one or more digital processors or processing circuits ("processors") and one or more storage devices (e.g., database 106 or other storage devices). In one embodiment, one or more servers and / or clients Different components of may perform different blocks or other portions of method 300. In some examples, a first device is described as performing blocks of method 300. An embodiment may have one or more blocks of method 300 performed by one or more other devices (e.g., other client devices or server devices) that may send results or data to the first device.

[0079] In one embodiment, the method 300, or portions thereof, may be initiated automatically by a system. In one embodiment, the implementing system is the first device. For example, the method (or portions thereof) may be performed periodically or may be initiated upon one or more specific events or conditions, e.g., an application being started by a user, The image is selected by the user and / or the settings are read by the method. In one embodiment, such conditions may be specified by the user in the user's stored custom preferences.

[0080] In one example, the client device may be a camera, a mobile phone, a smart phone, a tablet computer, a wearable device, or other client device capable of receiving content input (e.g., image capture) and user input (e.g., using a touch screen, using gestures, using a mouse or other pointing device, using a keyboard, etc.) by a user to the client device and capable of performing method 300. In another example, a client device or a server device may perform method 300. An embodiment may initiate method 300 based on user input. A user (e.g., an operator or end user) may, for example, select to initiate method 300 from a user interface, such as an application user interface or other user interface. In one embodiment, method 300 may be performed by a client device. In one embodiment, method 300 may be performed by a server device.

[0081] At block 302, in the implementation of method 300, it is checked whether user consent (e.g., user permission) has been obtained to use user data. For example, user data may include images captured by the user using a client device, e.g., images stored or accessed by the user using a client device, image metadata, user data related to use of image sharing applications, user data related to use of messaging applications, user data related to use of financial applications, user data related to use of social networking applications, user preferences, user biometric information, user characteristics (such as identity, name, age, gender, occupation, etc.), information about the user's social network and contacts, social and other types of actions and activities, content created or submitted by the user, ratings, and opinions, the user's current location, historical user data, images generated, received, and / or accessed by the user ... preferences, user preferences, user preferences, user preferences, user preferences, user preferences, user preferences, user preferences, user preferences, user preferences, user preferences, user preferences, user preferences, user preferences, user preferences, user preferences, user preferences, user preferences, user preferences, user preferences, user preferences, user preferences, user preferences, user preferences, user preferences, user preferences, user preferences, user preferences, user preferences, user preferences, user preferences, user preferences, user preferences, user preferences, user preferences, user preferences, user preferences, user preferences, user preferences, user preferences, user preferences, user preferences, user preferences, user preferences, user preferences This may include images viewed or shared, etc. One or more blocks of the methods described herein may, in some embodiments, use such user data.

[0082] If user consent has been obtained from an associated user whose user data may be used in method 300, then at block 304 it is determined that the blocks of the method herein may be performed with possible uses of the user data as described with respect to those blocks, and the method continues to block 312. If user consent has not been obtained, then it is determined at block 306 that the blocks should be performed without the use of user data, and the method continues to block 312. In one embodiment, if user consent has not been obtained, then the blocks are performed without the use of user data, using synthetic data and / or generic or publicly accessible data. The method is performed using available and publicly available data. If user consent has not been obtained, method 300 is not performed.

[0083] At block 312 of method 300, contextual information indicative of a target software application is determined. For example, when method 300 is implemented as part of an image library application that allows a user to generate image-based creations, such as image albums, image collages, videos, or printed publications, the contextual information may include the type of image creation.

[0084] In another example, when method 300 is implemented as part of an image library application that supports interaction with other applications, e.g., via an application programming interface (API), determining the context information includes determining the application type of the application invoking the image library application, e.g., an image sharing application (e.g., a standalone image sharing application, a social networking application, etc.), a financial application (e.g., an expense management application, an invoicing application, etc.), a messaging application (e.g., a chat or instant messaging application, an email application, a collaboration application, etc.). In this example, the application invoking the image library application is referred to as target software.

[0085] Additionally, in one embodiment, determining the context information may include receiving from the target software application a number of images to be selected, e.g., one image, two images, etc. In another example, such context information may include the type of image to be selected, e.g., an image with motion (e.g., a video clip), an image with a particular aspect ratio and / or orientation (e.g., square, portrait, landscape, etc.), etc. etc.), still images, video images, etc.

[0086] In another example, if user consent is obtained, determining the context information may include receiving identities (e.g., user identifiers such as usernames, login IDs, etc.) of participants in a messaging conversation (e.g., email recipients), characteristics of previous images shared via an image sharing application (e.g., scenery images, food images, self-images, etc.), etc. In another example, the context information may include user activities performed using a target software application, such as a financial application, for example, submitting an expense report, generating an invoice, etc.

[0087] In one embodiment, the target software application can provide the context information via an API. In one embodiment, if user consent is obtained, the context information can be provided as one or more image characteristics, e.g., “selfie,” “receipt,” “vacation,” etc., that can be matched with image characteristics associated with images in the user’s image library. Such image characteristics can be based on image content (e.g., depicted image features) and / or image metadata, if user consent is obtained. The context information may be based on context data (e.g., time of capture, location of capture, attributes and settings of the camera that captured it, etc.). In one embodiment, the context information may be provided as semantic concepts, e.g., "expense report", "chat conversation with Andrew", etc.

[0088] When the user provides consent, the context information may include application context from the target software application. For example, the application context may include an overview of a conversation or semantic concepts related to a conversation taking place over a messaging application, a time period for an expense report being prepared in a financial application, etc. For example, the application context may include "this conversation is about vacation in Bali," "expense report for November 2017," etc. Block 312 may be followed by block 314.

[0089] At block 314, one or more first images are identified based on the context information. For example, the first images can be based on matching the context information with image characteristics of images in an image library. In one embodiment, the matching can be based on an application type of the target application. For example, at block 314, one or more first images are identified based on the context information. For example, the first images can be based on matching the context information with image characteristics of images in an image library. In one embodiment, the matching can be based on an application type of the target application. Prior to 314, certain image characteristics may be associated with certain types of applications. For example, if the application type is a "financial application," images associated with labels indicating that the image is a "receipt," "document," "screenshot," etc. may be selected as the first image or images, and images with other characteristics (e.g., labels), such as "photo," "video," etc., may be excluded.

[0090] In one embodiment, the first image is identified by matching image characteristics and / or semantic concepts specified in the context information to image characteristics of images in the image library. For example, if the application type is a messaging application and the context information includes identities of participants in the messaging conversation, e.g., if the context information specifies that a chat conversation provided by the messaging application includes users Andrew and Joyce, an image depicting at least one of Andrew and Joyce may be selected as the first image.

[0091] In another example, if the application context specifies that a chat conversation provided by a messaging application is about a vacation in Bali, one or more semantic concepts are determined based on the application context. For example, the semantic concepts may include "vacation," "Bali," "beach," "Indonesia," "temple," etc. An image associated with a semantic concept determined from or specified within the application context may be selected as the first image. In another example, if the context information specifies an expense report for November 2017, an image of a receipt having an associated timestamp corresponding to November 2017 may be selected as the first image. Block 314 may be followed by block 316.

[0092] At block 316, a user interface including the first images is displayed. For example, the user interface may be displayed by any of client devices 120-124. The user interface allows a user to select one or more of the first images, similar to the user interface described above with reference to block 218. Block 316 may be followed by block 318.

[0093] At block 318, a user selection of one or more of the displayed images (e.g., a first image) is received. For example, block 318 may be similar to block 212 of Figure 2. Block 318 may be followed by block 320.

[0094] In block 320, image characteristics of the image selected in block 318 are determined, for example, in a manner similar to that described above in block 214. Block 320 may be followed by block 322.

[0095] At block 322, one or more additional images are identified. In one embodiment, the additional images may be identified based on the image characteristics determined at block 320. For example, if the user selection corresponds to an image associated with the labels "Bali" and "Andrew," an additional image having at least one of the labels is identified. In one embodiment, identifying the additional images is based on the image characteristics and context information determined at block 320, e.g., similar to block 314 described above. Block 322 may be followed by block 324.

[0096] At block 324, the user interface is updated to include one or more additional images. For example, the one or more additional images may be displayed in place of or in place of the first image. 2, block 218. Block 324 may be followed by block 326. In one embodiment, the updated user interface may include a section containing images selected by the user, such as a "selection tray" section. The "selection tray" may be separate from other portions of the user interface that contain the first image and one or more additional images available for selection by the user. Other features may be provided similar to those described with respect to block 218 of FIG. 2. Block 324 may be followed by block 326.

[0097] At block 326, it is determined whether an additional image selection has been received from the user, e.g., from an image displayed in an updated user interface. If an additional image selection has been received, the method proceeds to block 320 where image characteristics of the additional image selected by the user are determined. Blocks 320-326 may be repeated to provide additional image selection suggestions to the user. If an additional image selection has not been received, the method proceeds to block 328.

[0098] At block 328, the image selected by the user is provided to the target software application. For example, if the target software application is an image library application, the selected image may be provided as an image to be utilized in an image-based creative work. In another example, if the target software application is an image sharing or messaging application, the selected image may be provided such that the image is available for transmission (e.g., over a network) to one or more other devices via the image sharing or messaging application. In an embodiment, the selected image may be processed, e.g., automatically enhanced, cropped, compressed, converted to a different format, etc., prior to providing the image to the target software application. In an embodiment, the processing of the image may be based on contextual information received from the target software application.

[0099] Although method 300 has been described with reference to various blocks in FIG. 3, it should be understood that the techniques described in this disclosure may be performed without performing some of the blocks in FIG. 3. In an embodiment, one or more of the blocks shown in FIG. 3 may be combined. For example, two or more of blocks 314, 316, and 318 may be combined. In another example, two or more of blocks 320, 322, and 324 may be combined. In various embodiments, some of the blocks of method 300 may be performed in parallel or in a different order than that shown in FIG. 3.

[0100] In one embodiment, method 300 is performed by a client device, e.g., one or more of client devices 120-124. In one embodiment, method 300 is performed by a service device, e.g., server device 104. In one embodiment, method 300 is performed by a combination of client and server devices. For example, in one embodiment, blocks 302, 304, 306, 312, 318, 326, and 328 are performed by a client device, and blocks 314, 316, 320, 322, and 324 are performed by a server device. For example, such an approach may be useful when client device 120 has limited capabilities, e.g., to determine image characteristics by programmatically analyzing images, has limited storage capacity for storing image data, etc.

[0101] 4 is a schematic diagram of an exemplary user interface 400 including image selection suggestions, according to one embodiment. In various embodiments, the user interface 400 may be displayed by a display device, such as, in one embodiment, by a display screen of the client devices 120, 122, 124 and / or 126 of FIG. 1, or by a server system. The display may be displayed by the system 102.

[0102] As shown in FIG. 4, a portion of a user's image library is displayed in a user interface. In the example shown in FIG. 4, nine images from the image library are shown. Although FIG. 4 shows three rows of images with three square images per row, any number of rows and columns in the grid can be used based on available screen space. Additionally, different embodiments may display images in other configurations or arrangements. In some embodiments, more or fewer images from the library may be shown. In some embodiments, the images may be organized in a particular order, for example, reverse chronological order, based on timestamps associated with the images.

[0103] In the example shown in FIG. 4, the user selected the left-most image in the top row, as indicated by the check mark 402 on the image. As can be seen in FIG. 4, the image selected by the user (e.g., the first image) is a photo ID ("Jane Doe (pseudonym for an unidentified female)"). Using the techniques described herein, one or more suggested image selections (e.g., the second image) based on the user selected image are displayed in the user interface. For example, the suggested image selections are indicated by corresponding check marks 404, 406, and 408 on three other images in the user interface. The suggested image selections correspond to an image of a photo ID (John Doe (pseudonym for an unidentified male from XYZ Corporation) and two images of a receipt. The suggested image selections in this example are based on matching image characteristics of the first image (e.g., "document", "identity card") with those of other images in the image library.

[0104] User interface 400 further illustrates a suggested action element 410. In the example illustrated in FIG. 4, the suggested action is "select document and receipt." When the user selects suggested action element 410, other images from the user's image library that are documents and receipts are automatically selected based on the associated image characteristics. For example, automatically selected images may be displayed with a check mark similar to check marks 404, 406, and 408 when such images are scrolled or otherwise commanded via user input to be displayed in view of user interface 400. In one embodiment, such automatically selected images may be displayed in a display area of ​​the user interface separate from the display of images in the image library. In one embodiment, other or additional actions, such as "archive selected image," "auto-enhance selected image," "delete selected image," etc., may be indicated within the suggested action element.

[0105] In one embodiment, the suggested action may be based on image characteristics, e.g., different specific suggested actions may be associated with each specific image characteristic. For example, if a user selects a blurry image, the suggested action may be to delete the image. In another example, if a user selects a dark image, the suggested action may be to apply brightness or contrast enhancement. In one embodiment, the suggested action may be a composite action, e.g., "select an image, apply an image filter, and share the image via a messaging application." When a user selects a suggested action element 410, the corresponding action is automatically performed.

[0106] The user interface 400 allows a user to select images with reduced effort and reduced consumption of device resources, including memory, processing resources, and power. For example, instead of a user having to manually select each blurred image with many scrolling and searching operations on the device, the user interface allows the user to simply and may enable a user to accept the proposed selections presented with reduced device manipulation. Additionally, other images in an image library that are not displayed in user interface 400 but have characteristics that match the image selected by the user may be selected without the user having to scroll through the images and manually select each image.

[0107] 5A is a schematic diagram of an exemplary user interface 500 for providing image selection suggestions, according to one embodiment. In various embodiments, the user interface 500 may be displayed by a display device, such as, in one embodiment, a display device of the client devices 120, 122, 124 and / or 126 of FIG. 1 or the server system 102. It may be displayed by a display screen.

[0108] As shown in FIG. 5A, the user interface 500 includes a selection tray containing a user selected image 502 and two blank spots to indicate that additional images may be selected and added to the tray. The user interface 500 includes images from which the user may select. For example, the user may select one or more recent images from a "recent" section (504) of the user interface. In one embodiment, the images in the "recent" section may be organized in descending order of time based on capture timestamps associated with the images in the user's image library.

[0109] User interface 500 further includes suggestions 506 that include suggested image selections determined using the techniques described herein. In the example shown in FIG. 5, the suggested image selections are displayed in the form of a "Suggestions" card, overlaid on top of the "Recent" section 504. In different embodiments, the suggested image selections may be displayed in different sections of the user interface, may use a checkmark or other visual indicator to indicate the suggested image selections, etc. In one embodiment, the suggested image selections may be displayed in a preferred section of the user interface, for example, near the selection tray.

[0110] In the example shown in FIG. 5A, the user selected image 502 of a flower. Based on the user selection, other images in the user's image library that have characteristics matching one or more characteristics of image 502 are included in suggestions 506. For example, other images of flowers captured at locations similar to the location associated with image 502 may be displayed, such as images 512, 514, and 516. In the example shown in FIG. 5, a summary based on the matching characteristics, "Flowers on a Forest Path," is shown as section header 510 in the suggestions card. A second section with header "Friends on a Forest Path" (520) includes additional image suggestions of images 522, 524, and 526. The additional image suggestions are based on the associated location being "Forest Path." In different embodiments, any number of image suggestions may be provided. The user interface 500 allows the user to select from the recent section 504 and / or the suggestions card. 506. The user is then able to select one or more images from the selection bar 506.

[0111] In one embodiment, the image suggestions may be organized (e.g., grouped) in the user interface based on the degree of match with an image already selected by the user, e.g., image 502. For example, images 512-516 are shown in a first portion of the suggestion card 506 because they contain similar subject matter as selected image 502 and were captured on a forest path. Images 522-526 are shown in a second portion of the suggestion card 506 because the images were captured on a forest path, but the images do not depict flowers and therefore differ from selected image 502.

[0112] 5B is a schematic diagram of an exemplary user interface 530 that provides image selection suggestions, according to one embodiment. 5. After selecting images 522 and 512 via user interface 500, additional selected images may be displayed. Additional selected images are shown as part of a selection tray. Images 522 and 512 are removed from the suggestion card. Additional images 528 are added to the suggestion card, for example, based on matching image characteristics of images in an image library with the selected images 502, 512, and 522.

[0113] 5C is a schematic diagram of an example user interface 540 that provides image selection suggestions, according to an embodiment. As shown in FIG. 5C, an image-based creation, e.g., picture collage 542, is generated based on images selected by a user, e.g., images 502, 512, and 522. User interface 540 includes a suggestion card having one or more suggested images. For example, user interface 540 may enable a user to select one or more of the suggested images and add the images to the picture collage, e.g., by dragging and dropping the images from the suggestion card to the picture collage.

[0114] 6 is a block diagram of an example device 600 that may be used to implement one or more features described herein. In one example, device 600 may be used to implement a client device, e.g., any of client devices 115 shown in FIG. 1. Alternatively, device 600 may implement a server device, e.g., server 101. In an embodiment, device 600 may be used to implement a client device, a server device, or both a client device and a server device. Device 600 may be any suitable computer system, server, or other electronic or hardware device as described above.

[0115] One or more methods described herein may be implemented as a standalone program that may run on any type of computing device, a program running on a web browser, a mobile application ("app") running on a mobile computing device (e.g., a mobile phone, a smartphone, a tablet computer, a wearable device (watch, armband, jewelry, headwear, virtual reality goggles or glasses, augmented reality goggles or glasses, head mounted displays, etc.), a laptop computer, etc.). In one example, a client / server architecture may be used, e.g., the mobile computing device (as a client device) sends user input data to a server device and receives final output data for output (e.g., for display) from the server. In another example, all computations are performed by the mobile app (and / or other apps) on the mobile computing device. In another example, computation may be divided between the mobile computing device and one or more server devices.

[0116] In one embodiment, device 600 includes a processor 602, a memory 604, and an input / output (I / O) interface 606. Processor 602 includes one or more processors and / or a processor module that executes program code and controls the basic operation of device 600. or a processing circuit. A "processor" includes any suitable hardware system, mechanism, or component that processes data, signals, or other information. A processor may be a general-purpose central processing unit (CPU) having one or more cores (e.g., in a single-core, dual-core, or multi-core configuration), multiple processing units (e.g., in a multiprocessor configuration), a graphics processing unit (GPU), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a complex programmable logic device (CPLD), a dedicated circuit for achieving a function, a dedicated processor for performing neural network model-based processing, a neural circuit, a processor optimized for matrix calculations (e.g., matrix multiplication), or other system. , and a system with a processor 602. In an embodiment, the processor 602 may include one or more co-processors that perform neural network processing. In an embodiment, the processor 602 may be a processor that processes data to generate a probabilistic output, e.g., the output generated by the processor 602 may be imprecise or may be accurate within a range from an expected output. The processing need not be limited to a particular geographic location or have a time restriction. For example, the processor may perform its functions in "real-time", "offline", "batch mode", etc. Parts of the processing may be performed at different times and in different locations by different (or the same) processing systems. The computer may be any processor in communication with a memory.

[0117] Memory 604 is typically provided within device 600 for access by processor 602 and may be any suitable processor-readable storage medium, such as random access memory (RAM), read-only memory (ROM), electrically erasable read-only memory (EEPROM), flash memory, etc., suitable for storing instructions for execution by the processor. It may also be separate from and / or integrated with processor 602. The memory 604 may store software operated by the processor 602 on the server device 600, including an operating system 608, a machine learning application 630, other applications 612, and application data 614. The other applications 612 may include applications such as a data display engine, a web hosting engine, an image display engine, a notification engine, a social networking engine, etc. In an embodiment, the machine learning application 630 and the other applications 612 may each include instructions that enable the processor 602 to perform the functions described herein, e.g., some or all of the methods of FIGS. 2 and 3.

[0118] Other applications 612 may include, for example, an image editing application, a media display application, a communication application, a web hosting engine or application, a mapping application, a media sharing application, etc. One or more methods disclosed herein may operate in a number of environments and platforms, for example, as a standalone computer program that may run on any type of computing device, as a web application having a web page, as a mobile application ("app") running on a mobile computing device, etc.

[0119] In various embodiments, the machine learning application may utilize a Bayesian classifier, a support vector machine, a neural network, or other learning techniques. In one embodiment, the machine learning application 630 may include a trained model 634, an inference engine 636, and data 632. In one embodiment, the data 632 may include training data, e.g., data used to generate the trained model 634. For example, the training data may include any type of data, such as text, images, audio, video, etc. The training data may be obtained from any source, e.g., a data repository specifically marked for training, data for which permission has been provided for use as training data for machine learning, etc. In embodiments in which one or more users authorize the use of their respective user data to train a machine learning model, e.g., the trained model 634, the training data may include such user data. In embodiments in which multiple users authorize the use of their respective user data, the data 632 may include images (e.g., photographs or other user-generated images), communications (e.g., emails, text, Chat data (e.g. messages, voice, video, etc.), documents (e.g. spreadsheets, templates, It may contain any permitted data such as text documents, presentations, etc.

[0120] In some embodiments, data 632 may include collected data such as map data, image data (e.g., satellite imagery, overhead imagery, etc.), game data, etc. In some embodiments, training data may include synthetic data generated for training purposes, such as data not based on user input or activity in the context being trained, e.g., data generated from simulated conversations, computer-generated images, etc. In some embodiments, machine learning application 630 excludes data 632. For example, in these embodiments, trained model 634 may be generated, for example, on a different device and provided as part of machine learning application 630. In various embodiments, trained model 634 may be provided as a data file that includes a model structure or form and associated weights. Inference engine 636 may read the data file for trained model 634 and implement a neural network with node connectivity, layers, and weights based on the model structure or form specified in trained model 634.

[0121] The machine learning application 630 also includes a trained model 634. In an embodiment, the trained model may include one or more model forms or structures. For example, the model form or structure may include any type of neural network, such as a linear network, a deep neural network implementing multiple layers (e.g., "hidden layers" between an input layer and an output layer, where each layer is a linear network), a convolutional neural network (e.g., a network that divides input data into multiple portions or tiles, processes each tile separately using one or more neural network layers, and aggregates the results obtained from the processing of each tile), a sequence-to-sequence neural network (e.g., a network that takes sequential data as input, such as words in a sentence, frames in a video, etc., and produces a result sequence as output). The model form or structure may specify the connectivity between various nodes and the organization of the nodes into layers. For example, nodes in a first layer (e.g., an input layer) may receive data as input data 632 or application data 614. Such data may include, for example, one or more pixels per node if the trained model is used for image analysis. Subsequent intermediate layers may receive as input the output of the nodes of the previous layer, per the connectivity specified in the model format or structure. These layers are sometimes referred to as hidden layers. The final layer (e.g., the output layer) generates the output of the machine learning application. For example, the output can be a set of labels for an image, a representation of the image that allows comparison of the image with other images (e.g., the image's feature vector), an output sentence in response to an input sentence, one or more categories of input data, etc., depending on the particular trained model. In one embodiment, the model format or structure also specifies the number and / or type of nodes in each layer.

[0122] In different embodiments, the trained model 634 may include multiple nodes arranged in layers per model structure or morphology. In one embodiment, a node may be, for example, a memoryless computational node configured to process a unit of input and generate a unit of output. The computation performed by the node may include, for example, multiplying each of the multiple node inputs by a weight, obtaining a weighted sum, and adjusting the weighted sum with a bias or intercept value to generate the node output. In one embodiment, the computation performed by the node may also include applying a step / activation function to the adjusted weighted sum. In one embodiment, the step / activation function may include using a non-linear function. In various embodiments, such computations may include operations such as matrix multiplication. In one embodiment, computations by multiple nodes may be performed using, for example, multiple processor cores of a multi-core processor, individual processing units of a GPU, or dedicated A model may be implemented in parallel using neural circuits for processing inputs. In an embodiment, the node may include memory, e.g., to store and use one or more previous inputs in processing a subsequent input. For example, a node with memory may include a long short-term memory (LSTM) node. An LSTM node may use memory to maintain a "state" that allows the node to operate like a finite state machine (FSM). A model with such nodes may be useful in processing sequential data, e.g., words of a sentence or paragraph, frames of video, speech or other audio, etc.

[0123] In an embodiment, the trained model 634 may include embeddings or weights for individual nodes. For example, the model may begin as a number of nodes organized into layers specified by the model type or structure. At initialization, respective weights may be applied to the nodes connected per model type, e.g., to the connections of each pair of nodes in successive layers of a neural network. For example, each weight may be assigned randomly or may be initialized to a default value. The model may then be trained, e.g., with data 632, to generate results.

[0124] For example, training may include applying supervised learning techniques. In supervised learning, the training data may include multiple inputs (e.g., a set of images) and corresponding expected outputs for each input (e.g., one or more labels for each image). Based on a comparison of the model's output to the expected outputs, the values ​​of the weights are automatically adjusted, for example, to increase the probability that the model will generate the expected output given similar inputs.

[0125] In an embodiment, the training may include applying unsupervised learning techniques. In unsupervised learning, only input data may be provided and a model may be trained to distinguish the data, for example to cluster the input data into groups, each group containing more or less similar input data. For example, a model may be trained to distinguish images, such as distinguishing abstract images (e.g., synthetic images, human-drawn images, etc.) from natural images (e.g., photographs). The model may determine one or more image characteristics based on image content data and / or image metadata. The determined image characteristics may be trained to determine image labels and / or It may include semantic concepts.

[0126] In another example, a model trained using unsupervised learning may cluster words based on their use in an input sentence. In an embodiment, unsupervised learning may be used to generate a knowledge representation that may be used, for example, by machine learning application 630. In various embodiments, the trained model includes a set of weights or embeddings that correspond to the model structure. In embodiments in which data 632 is omitted, machine learning application 630 may include trained model 634 that is based on previous training, e.g., by a developer of machine learning application 630, by a third party, etc. In an embodiment, trained model 634 may include a set of weights that are fixed, e.g., downloaded from a server that provides the weights.

[0127] The machine learning application 630 also includes an inference engine 636. The inference engine 636 is configured to apply the trained model 634 to data, such as the application data 614, to provide inferences. In one embodiment, the inference engine 636 may include software code executed by the processor 602. In one embodiment, the inference engine 636 may include circuitry (e.g., for a programmable processor, field programmable logic) that enables the processor 602 to apply the trained model. In one embodiment, the inference engine 636 may include software instructions, hardware instructions, or a combination. In one embodiment, the inference engine 636 may be used by the operating system 608 and / or other applications 612 to call the inference engine 636. The model may provide an application programming interface (API) that can be called, for example, to apply the trained model 634 to application data 614 to generate inferences.

[0128] The machine learning application 630 may provide several technical advantages. For example, when the trained model 634 is generated based on unsupervised learning, the trained model 634 may be applied by the inference engine 636 to generate a knowledge representation (e.g., a numerical representation) from input data, e.g., the application data 614. For example, a model trained for image analysis may generate a representation of an image having a smaller data size (e.g., 1 KB) than the input image (e.g., 10 MB). In an embodiment, such a representation may help reduce the processing cost (e.g., computational cost, memory usage, etc.) to generate an output (e.g., a label, classification, a statement describing the image, etc.). In an embodiment, such a representation may be provided as an input to a different machine learning application that generates an output from the output of the inference engine 636. In an embodiment, the knowledge representation generated by the machine learning application 630 may be provided, for example, over a network to a different device for further processing. In such an embodiment, providing a knowledge representation rather than an image may provide a technical advantage, for example, enabling faster data transmission at reduced cost. In another example, a model trained to cluster documents may generate document clusters from input documents that are suitable for further processing (e.g., determining whether a document is related to a topic, determining a taxonomic category of a document, etc.) without needing to access the original documents, thus saving computational costs.

[0129] In some embodiments, the machine learning application 630 may be implemented in an offline manner. In these embodiments, the trained model 634 may be generated in a first stage and provided as part of the machine learning application 630. In some embodiments, the machine learning application 630 may be implemented in an online manner. For example, in such an embodiment, an application that invokes the machine learning application 630 (e.g., the operating system 608, one or more of the other applications 612) may utilize the inferences generated by the machine learning application 630 and, for example, provide the inferences to a user and generate a system log (e.g., actions taken by the user based on the inferences, if the user permits, or results of further processing, if utilized as input for further processing). The system log may be generated periodically, e.g., hourly, monthly, quarterly, and may be used with the user's permission to update the trained model 634, e.g., to update the embeddings of the trained model 634.

[0130] In an embodiment, the machine learning application 630 may be implemented in a manner that may be adapted to the particular configuration of the device 600 on which the machine learning application 630 is executed. For example, the machine learning application 630 may determine a computation graph that utilizes available computational resources, e.g., the processor 602. For example, if the machine learning application 630 is implemented as a distributed application on multiple devices, the machine learning application 630 may determine the computations to be performed on each device to optimize the computation. In another example, the machine learning application 630 may determine that the processor 602 includes a GPU with a particular number (e.g., 1000) of GPU cores and may accordingly (e.g., as 1000 individual processes or threads) configure the inference engine 602 to run on the GPU cores. can be implemented.

[0131] In some embodiments, the machine learning application 630 may implement a collection of trained models. For example, the trained models 634 may include multiple trained models, each applicable to the same input data. In these embodiments, the machine learning application 630 may select a particular trained model based on, for example, available computational resources, success rates with previous inferences, etc. In some embodiments, the machine learning application 630 may implement an inference engine 636 such that multiple trained models are applied. In these embodiments, the machine learning application 630 may combine the outputs from applying the individual models, for example, using a voting technique that scores the individual outputs from applying each trained model, or by selecting one or more particular outputs. Additionally, in these embodiments, the machine learning application may determine a time threshold (e.g., 0.5 ms) for applying the individual trained models. ) and utilize only those individual outputs that are available within the time threshold. Outputs that are not received within the time threshold may not be utilized, e.g., discarded. For example, such an approach may be appropriate when there is a time limit between invoking a machine learning application, e.g., specified by the operating system 608 or one or more applications 612.

[0132] In different embodiments, the machine learning application 630 can generate different types of output. For example, the machine learning application 630 may provide a representation or cluster (e.g., a numerical representation of the input data), a label (e.g., for the input data including images, documents, etc.), a phrase or sentence (e.g., describing the image or video, suitable for use as a response to the input sentence, etc.), an image, audio, or video (e.g., generated by the machine learning application in response to the input) (e.g., in response to an input video, the machine learning application 630 may generate an output video rendered in the style of a comic book or a particular artist, if the trained model 634 has been trained using training data from a comic book or a particular artist, etc.). In one embodiment, the machine learning application 630 may generate the output based on a format defined by a calling application, such as the operating system 608 or one or more applications 612. In one embodiment, the calling application may be another machine learning application. For example, such a configuration can be used in a generative adversarial network, where a calling machine learning application is trained with output from machine learning application 630, or vice versa.

[0133] Alternatively, any software in memory 604 may be stored in any other suitable storage location or computer readable medium. and other connected storage devices) to store one or more images, messages, one or more classifications, an electronic encyclopedia, a dictionary, a thesaurus, a knowledge base, message data, grammars, user preferences, and / or other instructions and data used in the features described herein. Memory 604 and any other type of storage (magnetic disk, optical disk, magnetic tape, or other tangible medium) may be considered "storage" or "storage device."

[0134] The I / O interface 606 may provide functionality that allows the server device 600 to interface with other systems and devices. The interfaced devices may be included as part of the device 600 or may be separate. For example, a network communication device, a storage device (e.g., memory and / or database 106), and an input / output Devices can communicate through an I / O interface 606. In one embodiment, the I / O interface can include input devices (keyboards, pointing devices, touch screens, microphones, cameras, scanners, sensors, etc.) and / or Or it can be connected to interface devices such as output devices (display devices, speaker devices, printers, motors, etc.).

[0135] Some examples of interfaced devices that may be connected to the I / O interface 606 include devices that store content, e.g., images, videos, and / or other data as described herein. The display device(s) 620 may include one or more display devices 620 that may be used to display user interfaces of output applications, as described below. The display device(s) 620 may be connected via a local connection (e.g., a display bus) and / or may be connected to device 600 via a network connection and may be any suitable display device. Display device 620 may include any suitable display device, such as an LCD, LED, or plasma display screen, a CRT, a television, a monitor, a touch screen, a 3D display screen, or other visual display device. For example, display device 620 may be a flat display screen provided on a mobile device, multiple display screens provided in a goggle or headset device, or a monitor screen for a computing device.

[0136] The I / O interface 606 can interface with other input and output devices. Some examples include one or more cameras that can capture images. An embodiment can provide a microphone for capturing sound (e.g., as part of a captured image, a voice command, etc.), an audio speaker device for outputting sound, or other input and output devices.

[0137] For ease of explanation, FIG. 6 shows one block for each of the processor 602, memory 604, I / O interface 606, and software blocks 608, 612, and 630. These blocks may represent one or more processors or processing circuits, operating systems, memory, I / O interfaces, applications, and / or software modules. 600 may not have all of the components shown and / or may differ from those shown herein. Instead of or in addition to the components depicted, the system may have other components, including other types of components. Although some components are described as performing the blocks and operations as described in some implementations of this specification, any suitable component or combination of components of environment 100, device 600, a similar system, or any suitable processor or processors associated with such a system, may perform the described blocks and operations.

[0138] The methods described herein may be implemented by computer program instructions or code that may be executed on a computer. For example, the code may be implemented by one or more digital processors (e.g., microprocessors or other processing circuits) and may be stored in a computer program product that includes a non-transitory computer-readable medium (e.g., storage medium), such as a magnetic, optical, electromagnetic, or semiconductor storage medium, including semiconductor or solid-state memory, magnetic tape, removable computer diskettes, random access memory (RAM), read-only memory (ROM), flash memory, rigid magnetic disks, optical disks, solid-state memory drives, and the like. The program instructions may also be For example, a server (e.g., a distributed system and / or a cloud computing system) The method(s) may also be included in or provided as an electronic signal in the form of software as a service (SaaS) delivered from ). Alternatively, the method(s) may be implemented in hardware (such as logic gates) or a combination of hardware and software. Examples of hardware may be programmable processors (e.g., field programmable gate arrays (FPGAs), complex programmable logic devices), general purpose processors, graphics processors, application specific integrated circuits (ASICs), etc. The method(s) may be implemented as part of or a component of an application executing on the system, or as an application or software running in conjunction with other applications and the operating system.

[0139] Although the description has been set forth with respect to specific implementations thereof, these specific implementations are exemplary only and not limiting, and the concepts illustrated in the examples may be applied to other examples and implementations.

[0140] It should be noted that the functional blocks, operations, features, methods, devices, and systems described in this disclosure may be integrated or divided into different combinations of systems, devices, and functional blocks, as known to those skilled in the art. Any suitable programming language and programming techniques may be used to implement the routines of a particular implementation. Different programming techniques, e.g., procedural or object-oriented techniques, may be employed. The routines may be executed on a single processing device or multiple processors. Although steps, operations, or computations may be presented in a particular order, the order may be changed in different particular embodiments. In some embodiments, multiple steps or operations shown as sequential in this specification may be performed simultaneously.

Claims

1. 1. A method executed by at least one processor, comprising: Receiving a first user input indicating a selection of one or more first images in the image library; determining one or more first image characteristics of the one or more first images; and identifying one or more second images in the image library, each of the one or more second images associated with at least one second image characteristic that matches at least one of the one or more first image characteristics; The method further comprises displaying a user interface, the user interface including the one or more second images, the user interface enabling selection of the one or more second images; The method comprises: receiving a second user input indicating a selection of at least one second image of the one or more second images; determining a suggested action associated with the one or more first images and the at least one second image based on the one or more first images and the at least one second image, and causing the user interface to display a corresponding suggested action element; and performing the suggested action on the one or more first images and the at least one second image in response to a user selection of the corresponding suggested action element.

2. The method of claim 1 , wherein the suggested action is additionally performed on one or more additional images in the image library having the one or more first image characteristics and the at least one second image characteristic.

3. The method of claim 1 or 2, wherein the proposed action has previously been associated with one or more particular image characteristics of the one or more first images and the at least one second image.

4. receiving the user selection of the corresponding proposed action element; and in response to receiving the user selection, performing the suggested action associated with the corresponding suggested action element, the suggested action including one or more of: deleting the one or more first images and the at least one second image, or automatically enhancing the one or more first images and the at least one second image.

5. The method of claim 4, wherein the proposed actions further include one or more of archiving at least one of the one or more first images and the one or more second images, or generating an image-based creation that includes the at least one of the one or more first images and the one or more second images.

6. identifying one or more third images in the image library, each of the one or more third images associated with at least one third image characteristic that matches at least one of the one or more first image characteristics and the at least one second image characteristic; 6. The method of claim 1, further comprising causing an updated user interface to be displayed, the updated user interface including the one or more third images, the updated user interface enabling selection of the one or more third images.

7. 7. The method of claim 1, wherein determining the one or more first image characteristics comprises determining that at least one of the one or more first images is blurred, and identifying the one or more second images comprises identifying each of the one or more second images based on a degree of blurriness.

8. 8. The method of claim 1, further comprising determining a context of image selection, and identifying the one or more second images is based on the context of image selection, the context of image selection being the creation of an image-based creation, the image-based creation comprising at least one of an image album, an image collage, a video, or a printed publication.

9. 8. The method of claim 1, further comprising determining a context of image selection, wherein identifying the one or more second images is based on the context of image selection, the context of image selection being to provide an image to a target software application of a particular application type, the target software application comprising one or more of an image sharing application, a financial application, or a messaging application.

10. 10. The method of claim 1, wherein the one or more second images are a plurality of second images, and wherein causing the user interface to display comprises causing a first group of the plurality of second images to be displayed in a first portion of the user interface and causing a second group of the plurality of second images to be displayed in a second portion of the user interface, the first group having a higher degree of match with the one or more first images than the second group.

11. 11. The method of claim 10, wherein the first group includes one or more of the second images having characteristics that match both image feature characteristics and location characteristics of the one or more first images, and the second group includes one or more of the second images having characteristics that match only one of the image feature characteristics or the location characteristics of the one or more first images.

12. 12. The method of claim 1, wherein determining the one or more first image characteristics comprises determining a location associated with at least one of the one or more first images, and identifying the one or more second images comprises selecting images from the image library associated with respective locations that are within a threshold distance of the location associated with the at least one of the one or more first images.

13. The method of any one of claims 1 to 12, further comprising generating an image album, said image album including said one or more first images and said at least one second image.

14. The method of any one of claims 1 to 12, further comprising generating an image collage, said image collage including said one or more first images and said at least one second image.

15. A program for causing one or more processors to execute the method according to any one of claims 1 to 14.

16. A memory storing the program according to claim 15; and one or more processors for executing the program.

Images