Method and system for predicting potentially relevant topics

Abstract

Embodiments for predicting / determining topics potentially relevant to a user using digital media files retrieved and / or created by the user are described herein. In various embodiments, (multiple) digital media files created and / or retrieved by the user using a client device can be applied as input in a trained machine learning model to generate an output indicating an object detected in the digital media file, in some cases, the trained machine learning model is local to the client device. Data indicating the indicated object can be provided to a remote computing system without providing the digital media file itself. In some embodiments, information associated with the indicated object can be retrieved and actively output to the user. In some embodiments, the frequency with which the object appears in a corpus of digital media files can be considered when determining the likelihood that a detected object is potentially relevant to a user.

Description

Background Art

[0001] Information useful for controlling data sent to a device may be determined from a variety of different sources, such as documents (particularly cloud-based documents), calendars, explicit user preferences, etc. However, it may be desirable to control access to the source of the information. Summary of the invention

[0002] The present disclosure generally relates to methods, devices, and computer-readable media (transient and non-transient) for utilizing digital media files retrieved and created by users. In various embodiments, digital media files created / retrieved by users, such as digital photos, digital videos, digital audio files, etc., can be analyzed using various techniques to detect one or more objects represented in these digital media files. "Objects" can include anyone, anywhere, or anything. Some unique objects, such as specific people (e.g., celebrities), unique objects (e.g., infamous objects, such as the Hope Diamond), etc., may be alternatively referred to as "entities" here and / or elsewhere, particularly in the context of knowledge graphs. Most objects, such as fungible objects, may have types or classes, and / or may be members of various hierarchies (e.g., Granny Smith→Apple→Fruit→Plant→Lifestyle). Based on these detected objects, alone or in combination with other user interest signals, potentially relevant topics (e.g., user interests) can be determined and used to present relevant information to the user.

[0003] Various techniques may be implemented locally on a client device operated by a user to detect objects represented in a digital media file. Data indicative of these detected objects may be provided by the client device to a remote computing server (e.g., a cloud) without providing the underlying digital media files, so that the remote computing system may determine whether these detected objects demonstrate a subject matter that is likely to be relevant to the user. For example, in some embodiments, one or more trained machine learning models may be stored locally on the client device. Data indicative of digital media created / captured at the client device may be applied (locally at the client device) as input to a trained machine learning model to generate an output indicative of one or more objects. For example, such an output may be used as described above to proactively present content that is potentially relevant to the user at one or more client devices.

[0004] The techniques described herein produce various technical advantages. As an example, employing a local on-device model to detect objects in digital media files created and / or retrieved by a user enables a remote server to use information associated with the user's digital media files without exposing the user's digital media files (particularly digital media files created by the user that may be considered highly private) to the cloud. Further, using a local on-device model enables data to be sent by a client device for providing information related to potentially relevant topics, wherein the data consumes less network bandwidth than if the digital media files were sent instead. This may be beneficial, for example, in situations where the client device utilizes a limited bandwidth network connection and may enable relevant information to be provided to the client device more quickly (e.g., without having to wait for the client device to utilize a higher bandwidth network connection).

[0005] As another example, identifying topics potentially relevant to the user and using these topics to obtain / generate / display content presented to the user can reduce the input required by these users to reach their desired content. For example, sorting search results based on topics identified using the techniques described herein can allow users to find what they are looking for faster with less input, thereby saving computing resources, such as network bandwidth, memory, battery power, etc. To this end, in various embodiments, topics potentially relevant to the user can be associated with the user (e.g., by being associated with the user's profile) and persistently stored in a memory, such as at one or more client devices operated by the user and / or in the cloud, for future use. As another example, digital media files created by or at least edited by a user may be more indicative of the user's interests than digital media files that are only passively retrieved (e.g., viewed, played back, etc.) by the user, files shared with other users, or other signals (such as search history (which may indicate past interests, rather than more core interests)). Therefore, the techniques described herein can identify topics that are very likely to be relevant to the user, thereby improving the user experience.

[0006] In one aspect, a method implemented by one or more processors may include: obtaining one or more digital media files created by a user using a client device; applying the one or more digital media files as input to one or more trained machine learning models local to the client device, wherein the one or more trained machine learning models generate an output indicating one or more objects detected in each of the one or more digital media files; providing data indicating the indicated one or more objects to a remote computing system without providing the one or more digital media files; receiving information associated with the indicated one or more objects from the same remote computing system or a different remote computing system; and actively outputting information output associated with the indicated one or more objects to the user at the same client device operated by the user or a different client device.

[0007] In various implementations, providing may cause the remote computing system to formulate a search query based on the identified one or more objects, and the information associated with the indicated one or more objects includes information responsive to the formulated search query.

[0008] In various embodiments, the method may further include: receiving a search query from a remote computing system; and publishing the search query to receive information responsive to the search query in response to publishing. In various embodiments, the remote computing system may use the formulated search query itself to obtain information responsive to the formulated search query.

[0009] In various implementations, the remote computing system may determine one or more topics potentially relevant to the user based on data indicative of the indicated one or more objects, and information associated with the indicated one or more objects may include information associated with one or more interests in the database.

[0010] In another aspect, a computer-implemented method may be provided, the method comprising: receiving data indicating one or more digital media files from one or more client devices operated by a user; identifying one or more objects detected in each of the one or more digital media files based on the data indicating the one or more digital media files; determining the frequency of each of the one or more objects in a corpus of digital media files provided by a user population; selecting one or more of the objects based on the corresponding frequencies; identifying one or more topics potentially relevant to the user based on the selected one or more objects; formulating a search query based on the identified one or more topics; and actively causing information responsive to the formulated search query to be output at one or more of the client devices operated by the user.

[0011] In various implementations, the data indicating one or more digital media files may include one or more digital images, and identifying the one or more objects may include performing image recognition on the one or more digital images to identify the one or more objects. In various implementations, the data indicating one or more digital media files may include one or more audio files, and identifying the one or more objects may include performing audio analysis on the one or more audio files to identify the one or more objects.

[0012] In various embodiments, the data indicating the one or more digital media files may include one or more feature vectors or reduced dimension embeddings extracted from the one or more digital media files. In various embodiments, the selection may be further based on the location of the one or more objects in the one or more digital media files.

[0013] In various embodiments, the selection may be further based on a measure of the focal length of one or more objects in the one or more digital media files. In various embodiments, the selection may be further based on location coordinates associated with the one or more digital media files received from the one or more client devices. In various embodiments, the selection may be further based on whether the user created or retrieved the one or more digital media files received from the one or more client devices.

[0014] Embodiments can utilize digital media files retrieved and / or created by users to predict / determine topics potentially relevant to users (e.g., their interests or the interests of others close to them, user preferences, etc.). In various embodiments, information and / or data related to these topics can then be actively provided to users. In one aspect, objects detected in digital media files created / retrieved by users can be utilized to identify topics potentially relevant to users to varying degrees. For example, backgrounds and / or incidental objects (such as furniture, trees, scenery, etc.) detected in digital photos (and / or digital videos) of users may not be highly indicative of user interests, and the backgrounds and / or incidental objects are common among the corpus of digital photos acquired by user groups. In contrast, less common objects that are less encountered in the corpus of digital photos may be more highly indicative of user interests. This is particularly true if these less common objects are detected in multiple digital photos taken by users. For example, if bourbon bottles are not particularly common in the expected library of digital photos acquired by user groups, the fact that bourbon bottles feature significant features in digital photos of specific users may indicate the user's interest in bourbon or at least the interest in bourbon bottles. Alternatively, this fact may indicate the user's interest in decorative bottles or other unusual bottles. And if there are multiple photos of different types of bottles (e.g., bourbon bottles, scotch bottles, wine bottles, medicine bottles, etc.), this may indicate a general interest in bottles. As another example, if one or more bourbon bottles appear in the user's photos, but the user also views and / or creates pictures of the distillery, e.g., while traveling at the distillery, and / or views and / or creates pictures of bourbon labels, these may collectively be strong signals of a general interest in bourbon.

[0015] Once the topics potentially relevant to the user are identified using the techniques described herein, these identified topics can be used for various purposes. In some embodiments, information associated with the identified topics can be obtained and actively presented or "pushed" to the user, such as a notification "card" or "sticker" presented on the touch screen of a smartphone, as an output from an automated assistant for participating in a human-computer dialogue conference, an incoming text message, etc. using it. In some embodiments, topics potentially relevant to the user can be used to formulate search queries, for example, by a client device operated by the user or by a remote computing server forming what is commonly referred to as a "cloud". Information in response to the search query can be obtained and pushed to the user. Additionally or alternatively, in some embodiments, once it is determined that the topic is potentially relevant to the user, one or more user topics can be associated with various predetermined content (e.g., display content) that is automatically presented to the user in the database.

[0016] Embodiments such as these can make it possible to identify topics potentially related to users, and the users themselves may not know these topics. As an example, it is assumed that a father obtains many digital photos and / or digital videos of his family. It is further assumed that one of his children is always drawn in these data media files that interact with a specific type of object. For example, in one photo, his son is playing with a toy motorcycle, in another photo, his son is wearing a shirt depicting a motorcycle, and in another example, his son poses on his uncle's motorcycle, etc. Even if the father is not aware of his son's interest in motorcycles for some reason, the technology described herein can also be identified as being related to the father because the theme is the son's interest (even if the father himself is obviously not interested in motorcycles, his son's interest in motorcycles may at least make this theme related to the father.) Therefore, the information related to the motorcycle can be pushed to the father. For example, as the son's birthday approaches, the information about the son's gift potentially related to the motorcycle can be presented to the father.

[0017] Other embodiments may include a non-transitory computer-readable storage medium storing instructions executable by a processor to perform a method, such as one or more of the methods described above. Yet another embodiment may include a system including a memory and one or more processors operable to execute instructions stored in the memory to perform a method, such as one or more of the methods described above.

[0018] It should be understood that all combinations of the above concepts and additional concepts described in more detail herein are considered part of the subject matter disclosed herein. For example, all combinations of the claimed subject matter appearing at the end of this disclosure are considered part of the subject matter disclosed herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 Example embodiments are shown in which selected aspects of the disclosure may be implemented.

[0020] Figure 2 , Figure 3 and Figure 4 Depicted are example scenarios in which the techniques described herein may be employed, according to various implementations.

[0021] Figure 5 An example of how content may be actively output after performing the techniques described herein is schematically demonstrated.

[0022] Figure 6 Flowcharts illustrating example methods of performing selected aspects of the present disclosure are depicted.

[0023] Figure 7 Another flow chart illustrating an example method of performing selected aspects of the present disclosure is depicted.

[0024] Figure 8 An example architecture of a computer system is schematically depicted. DETAILED DESCRIPTION

[0025] Figure 1 An example environment is shown in which information can be determined by a user creating and / or retrieving (e.g., consuming) digital media files. The example environment includes a client device 106 and a cloud-based system 102. The cloud-based system 102 can be implemented in one or more computers that communicate, for example, over a network (not shown). The cloud-based system 102 can be an example of a system in which the systems, components, and techniques described herein can be implemented and / or the systems, components, and techniques described herein can be interfaced. Although described as being implemented largely on a "cloud-based system" herein, the disclosed technology may actually be performed on systems used for various purposes (such as email systems, text messaging systems, social networking systems, voicemail systems, productivity systems, enterprise software, search engines, etc.).

[0026] A user may interact with the cloud-based system 102 via a client device 106. Other computer devices may communicate with the cloud-based system 102, including, but not limited to, additional client devices and / or one or more servers implementing services of a website that has partnered with a provider of the cloud-based system 102. However, for the sake of brevity, examples are described in the context of a client device 106.

[0027] The client device 106 may be a computer that communicates with the cloud-based system 102 over a network, such as a local area network (LAN) or a wide area network (WAN), such as the Internet (one or more such networks generally indicated at 112). For example, the client device 106 may be a desktop computing device, a laptop computing device, a tablet computing device, a mobile phone computing device, a computing device of a user's vehicle (e.g., an in-vehicle communication system, an in-vehicle entertainment system, an in-vehicle navigation system), a stand-alone interactive speaker operating a voice-interactive personal digital assistant (also referred to as an "automated assistant"), or a user's wearable device that includes a computing device (e.g., a user's watch with a computing device, a user's glasses with a computing device, a wearable music player). Additional and / or alternative client devices may be provided.

[0028] Client device 106 may include various software and / or hardware components. Figure 1, client device 106 includes visual sensor 107 and microphone 108. Visual sensor 107 can take various forms, such as a digital camera, a passive infrared ("PIR" sensor), an RGBd sensor, a stereo camera, etc. Visual sensor 106 can be operated, for example, by a user (not shown), to capture one or more digital images, for example as independent digital images and / or as a sequence of digital image frames forming a digital video. Microphone 108 can be operated, for example, by the user to capture various sounds, such as the user's speech. In some scenarios, visual sensor 107 and microphone 108 can be used together, for example, to capture multimedia audio / visual digital media files, such as video recordings.

[0029] The client device 106 can also execute various software. Figure 1 In the illustrated embodiment, the client device 106 executes a media client 109 and an object recognition client 110. In various embodiments, the media client 109 may be operated, for example, by a user to create and / or retrieve (e.g., for consumption) digital media files. As used herein, "digital media files" may include digital images (e.g., photos), digital videos (with or without sound), computer-aided design ("CAD") files, audio files (e.g., songs, broadcast lessons, etc.), etc. In some embodiments, the client device 106 may execute multiple media clients 109, for example, for multiple different types of digital media files. However, for the sake of brevity and simplicity, the embodiments mentioned herein will be described as having a single media client 109 installed. In some embodiments, digital media files created using the media client 109 may be uploaded (e.g., backed up, transferred) to a cloud-based system 102, for example, for safe and / or reliable storage in one or more cloud storage databases 122 via an online storage engine 120. Additionally or alternatively, the online storage engine 120 may form part of a social network so that digital media files uploaded or shared by users on their social network accounts may be analyzed using the techniques described herein.

[0030] The object recognition client 110 may be configured to recognize objects represented in digital media files retrieved and / or created using the media client 109. In some embodiments, all object recognition may be performed at the cloud-based system 102, in which case the object recognition client 110 cannot be installed on the client device 106. However, in some embodiments, the object recognition client 110 may be installed on the client device 106 to protect the user's privacy. For example, in some embodiments, the object recognition client 110 may have access to an "on-device" object recognition model database 111 (the terms "index" and "database" are used interchangeably here). The on-device model database 111 may store one or more object identification modules and / or models, which may be used, for example, by the object identification client 110 to detect objects represented in digital media files and / or provide output indicating one or more detected objects. Therefore, data generated by these models, rather than the original data of the digital media files in which the objects are detected, may be provided to the cloud-based system 102. An example of how object recognition may be performed will be described below.

[0031] In some implementations, the client device 106 and the cloud-based system 102 each include memory for storing data and software applications, a processor for accessing data and executing applications, and components to facilitate communications over a network 112. The operations performed by the client device 106 and / or the cloud-based system 102 may be distributed across multiple computer systems. For example, the cloud-based system 102 may be implemented as a computer program running on one or more computers in one or more locations coupled to each other over a network.

[0032] In various embodiments, the cloud-based system 102 may include an online storage engine 120, an object recognition engine 124, an object frequency engine 128, a topic identification engine 132, and / or a knowledge graph engine 136. In some embodiments, one or more of the engines 120, 124, 128, 132, and / or 136 may be combined and / or omitted. In some embodiments, one or more of the engines 120, 124, 128, 132, and / or 136 may be implemented in a component separate from the cloud-based system 102. In some embodiments, one or more of the engines 120, 124, 128, 132, and / or 136, or any operable portion thereof, may be implemented in a component executed by the client device 106.

[0033] In this specification, the terms "database" and "index" are used interchangeably and broadly refer to any data set. The data of the database and / or index need not be structured in any particular way, and it may be stored on storage devices in one or more geographic locations. Thus, for example, indexes 111, 122, 126, 130, 134, 138 may include multiple data sets, each of which may be organized and accessed differently.

[0034] Turning now to the cloud-based system 102, in various embodiments, the online storage engine 120 may be configured to store various types of digital content (such as digital media files) of a user in a cloud storage 122, which is generally referred to as "cloud storage." For example, a user may have an online profile that synchronizes data between multiple client devices 106 operated by the user. Such synchronization may be facilitated by the online storage engine 120, for example, by storing a copy of all of the user's digital content to be synchronized in the cloud storage 122. The user may intentionally cause the online storage engine 120 to store digital content in the cloud storage 122, for example, by directly uploading such content and / or by storing such content in a storage folder on one or more client devices 106 designated for cloud storage. Additionally or alternatively, in some embodiments, the user's various digital content may be automatically stored (e.g., by the online storage engine 120) in the index 122 as part of an automatic backup process. For example, some cloud computing service providers provide their users with free or fee-based automatic backup of digital photos, videos, or other digital media files.

[0035] Like the object recognition client 110, the object recognition engine 124 may be configured to identify objects represented in a digital media file. However, the object recognition client 110 operates on a resource-constrained client device 106 and may therefore have limited capabilities. For example, in some embodiments, a limited number of on-device object recognition models may be stored in an on-device model index 111. In contrast, the object recognition engine 124 may be able to utilize the nearly unlimited resources of the cloud to perform object recognition, and therefore, may be far more capable, more robust, etc. than the object recognition client 110. For example, the object recognition engine 124 may have access to an index 126 of object recognition modules and / or models that is far larger than the on-device index 111 available to the object recognition client 110. In some embodiments, if the object recognition client 110 is unable to detect / recognize one or more objects in a digital media file, or at least is unable to recognize them with sufficient confidence, the digital media file (or some variation of data indicating it) may be provided to the object recognition engine 124 to perform object recognition analysis.

[0036] Object recognition and / or detection may be performed using a variety of different techniques, for example, by the object recognition client 110 and / or the object recognition engine 124. In some embodiments, the on-device model index 111 and / or the cloud-based model index 126 may include object models (e.g., based on computer-aided design or "CAD") that may be used and / or triggered to provide inferences about object types, for example, using visual and / or depth data obtained by one or more visual sensors (e.g., 107). Additionally or alternatively, the on-device model index 111 and / or the cloud-based model index 126 may include 2D patterns or profiles of objects that may be matched to portions of 2D image data (e.g., video frames) captured by one or more visual sensors (e.g., 107). In other embodiments, the on-device model index 111 and / or the cloud-based model index 126 may include routines (e.g., state machines) that may be implemented / triggered by the object recognition client 110 and / or the object recognition engine 124 to provide inferences about object types. In some embodiments, the object recognition client 110 and / or the object recognition engine 124 can detect textual content in a digital image, for example, by using optical character recognition (“OCR”) to identify text that is contained, for example, in the background of the image (for example), on a garment depicted in the image, and so on.

[0037] In some implementations, one or more machine learning models may be trained to generate outputs that indicate one or more objects detected in a digital media file. For example, one or more convolutional neural networks ("CNNs") may be trained to generate outputs based on digital image files (personal photos or video frames) that indicate one or more objects detected in a digital media file and, in some cases, confidences associated with those detected objects.

[0038] As a non-limiting example, a CNN can be trained using a corpus of digital images, each digital image being labeled with one or more objects represented in the digital image. Each training example can be used as an input to the CNN to generate an output. The output can then be compared to the label associated with the training example to determine the difference or error. Based on this error, various techniques can be used to train the CNN, such as back propagation, gradient descent, etc. Once trained (e.g., using hundreds or thousands of training examples), unlabeled digital images can be used as input to the CNN to generate an output indicating one or more objects detected in the digital image, and in some cases indicating a confidence measure of the output.

[0039] In addition to identifying objects in digital images (whether from personal digital photos or from digital video frames), in some embodiments, the object recognition client 110 and / or the object recognition engine 124 can be configured to detect objects (or more generally topics) in digital audio content (whether independent of digital video or accompanying digital video). For example, in some embodiments, the object recognition client 110 and / or the object recognition engine 124 can employ techniques such as speech recognition to detect mentions of objects in audio data. In some embodiments, the on-device model index 111 and / or the cloud-based model index 126 can store one or more models, such as one or more machine learning models, that are trained to detect mentions of various objects in audio data. Additionally or alternatively, in some embodiments, the object recognition client 110 and / or the object recognition engine 124 can employ speech-to-text processing to generate speech recognition output (i.e., text) from the audio data, and then analyze the text using techniques such as natural language processing, entity annotation, etc. to identify the mentioned objects, or more generally, to identify topics of potential interest to the user.

[0040] The subject identification engine 132 can be configured to identify one or more subjects potentially related to one or more users based on various signal identifications. In some embodiments, these subjects can be stored in the user interest index 134, for example, associated with applicable users, so that they can be used for various purposes later, including but not limited to caching information on the client device (for example, when the client device is connected to a high-bandwidth data connection), sorting search results, generating alternative query suggestions, selecting and presenting digital content, recommending media / restaurants, etc. These signals may take various different forms and may come from various different sources. In some embodiments, the subject identification engine 132 can identify subjects potentially of interest to or related to the user based on the analysis performed by the object recognition client 110 and / or the object recognition engine 124 on the digital media files created and / or retrieved by the user. For example, it is assumed that the object recognition engine 124 detects multiple representations of classic cars in multiple digital photos captured by the user. Even if the user does not explicitly express interest in classic cars, these detections can also be used by the subject identification engine 132 to determine that the user may be potentially interested in classic cars.

[0041] In addition to the unmistakable fact that an object has been detected in a digital media file, the topic identification engine 132 may also consider various other properties of the digital media file and the role or prominence of the detected object in the file. In some embodiments, the prominence of a detected object in a digital image may be determined (e.g., at 110 / 124) and used to weight the potential of the object as a topic relevant to the user. For example, the location of the detected object in the image, the size of the detected object relative to the entire image, the size of the detected object relative to other objects detected in the image, a measure of the focus of the object relative to other detected objects (e.g., how blurry the object is compared to other detected objects), the number of objects of a particular type in one or more images (e.g., three apples depicted in a picture may be more relevant than one apple), and other similar data points may be provided, for example, by the object recognition client 110 and / or the object recognition engine 124 and considered (e.g., by the topic identification engine 132) in determining a measure of the prominence of the detected object. In some embodiments, a measure of prominence for a particular detected object can be compared (e.g., by topic identification engine 132) to one or more thresholds and / or one or more other measures of prominence associated with other detected objects to determine whether (or to what extent) the object (and more generally, a topic determined from the object) should be considered potentially relevant to the user.

[0042] For example, if a particular type of car is prominently represented in one or more digital images (such as in the foreground, in the middle of the image, etc.), this may be highly probable that the detected car (and more generally the subject matter derived therefrom) may be of interest to the user. In contrast, if an object is detected in the background of a digital image, detected off to the side, or detected not even fully depicted in the image (e.g., cut off at the top, the side, etc.), this may be less probable that the user is potentially interested in the object, as the representation of the object in the digital media file may be merely accidental and / or incidental. In some embodiments, the subject identification engine 132 may take into account how many objects are detected in a particular digital media file, so that any individual object detected in a digital image is not overly heavily weighted. In some such embodiments, the prominence of a detected object in a digital image may be considered, for example, by the subject identification engine 132, in conjunction with the total number of objects in the image. For example, if a large number of objects are detected in a given digital image, but a particular object is far more prominently represented than other detected objects, then that particular object may be weighted more heavily by the subject identification engine 132.

[0043] Additionally or alternatively, in some embodiments, the theme identification engine 132 can identify based on metadata associated with the digital media content. For example, suppose a user gives a particular digital photo file name that describes the object represented in the digital photo. Such metadata can be used, for example, by the theme identification engine 132 as a signal that (alone or in combination with other signals) can suggest that the user is interested in one or more topics. In some embodiments, the fact that the user takes the time to explicitly define metadata identifying one or more topics may be highly evidence of the user's potential interest in those topics. For example, other available metadata can include folders in which digital media files are stored on the client device 106 and / or stored in the cloud storage 122. For example, if a large number of images are stored in a folder named "Pictures of Birds", the theme identification engine 132 can infer that the user is interested in the topic of birds. As another example, in some embodiments, the theme identification engine 132 can consider metadata associated with the media consumed (e.g., streamed) by the user. For example, the title, introduction, description, cast, crew, or other metadata associated with the online streaming video watched by the user can be used to identify topics that are potentially relevant to the user.

[0044] Additionally or alternatively, in some embodiments, the theme identification engine 132 can identify themes based on the nature of the user's interaction with one or more digital media files over time. For example, if a user tends to repeatedly open specific digital images in their library (more than other images in their library), the theme identification engine 132 can determine that the objects represented in those specific images are potentially relevant to the user, especially if other less viewed images do not tend to depict those same objects. This can include digital images stored in the application cache. For example, suppose a user repeatedly revisits a specific web page that lists cars for sale, and the web page includes a photo of the car. When this photo is loaded from the cache into the web browser (rather than re-downloading the photo), it may be considered that the user is viewing the photo again, which in turn causes the objects in the photo to be identified as potentially relevant.

[0045] In addition to or in lieu of objects detected in digital media files created / retrieved by the user, the topic identification engine 132 may employ various other signals in various embodiments. In some embodiments, signals such as the user's search query history, browsing history, explicitly provided personal preferences, etc. may be used by, for example, the topic identification engine 132 to identify one or more topics of potential interest to the user. Additionally or alternatively, in some embodiments, signals such as the content of the user's email / text messages, the content of the user's social media profile, the content of the user's calendar, and various other application data may be used by, for example, the topic identification engine 132 to identify topics potentially relevant to the user. In various embodiments, previously identified topics potentially relevant to the user may themselves be used to identify new potentially relevant topics. For example, suppose a user takes photos of skis, snowy peaks, ice skates, and views images of winter equipment (e.g., when shopping online). These may cause the user to be associated with relatively narrow topics such as "skiing", "mountaineering", "skating", and "winter equipment". In some embodiments, these topics may be collectively considered by, for example, the topic identification engine 132 to identify other topics higher in the topic hierarchy (e.g., as determined by the knowledge graph engine 136), such as "winter sports".

[0046] It should be understood that in various embodiments, users may have the option to opt-in or opt-out of various data being used as signals using the techniques described herein. For example, in some embodiments, a default user profile setting may be that few or no signals are available to the topic identification engine 132. Only when the user explicitly provides authorization for each signal (or group of signals) can the topic identification engine 132 gain access to these signals for topic identification purposes.

[0047] The subject identification engine 132 can weight various types of signals differently. For example, an object detected in a digital media file created by a user may be more likely to prove that the user is potentially interested in topics related to those objects than an object detected in a digital media file that the user only retrieves for consumption (e.g., viewing, playback, etc.). Similarly, in some embodiments, there may be a hierarchy of weights assigned to objects detected in digital media files that interact with the user. For example, the maximum weight can be assigned to an object detected in a digital media file created by a user. An intermediate amount of weight can be assigned to an object detected in a digital media file shared by, for example, another user with the user. The minimum amount of weight can be assigned to an object detected in a digital media file retrieved by a user over the Internet (e.g., as part of a web search). Other factors that may be considered as part of this "hierarchy" of weights include, but are not limited to, how long a user consumes a particular digital media file, whether a user "finishes" consuming a digital media file (e.g., playing a whole song or a whole video), whether a user has edited a digital media file, whether a user has shared a digital media file, and the like.

[0048] In some embodiments, various signals may be analyzed in combination to determine whether / how relevant a particular topic may be to a user. For example, in some embodiments, detection of an object in a user-created digital media file may not by itself be sufficient to warrant using the detected object to identify topics of potential interest to the user. However, if other signals corroborate the initial user-created digital media file signal, the detected object may be used to identify one or more topics of potential interest to the user.

[0049] In some implementations, signals that a user is potentially interested in a topic can be weighted based in part on whether objects associated with the topic are common in digital media content created / retrieved by the user community. Figure 1 In, for example, object frequency engine 128 can be configured to determine the frequency of one or more objects in a corpus of digital media files retrieved and / or created by a user community (e.g., stored in index 130 or even in index 122). In some such embodiments, the frequency with which an object appears in the corpus may be inversely proportional to how multiply the object is weighted when identifying topics of potential interest. For example, furniture such as desks or dining tables may be very common in a corpus of images. Thus, if a particular user takes one or more digital photographs that capture a dining table or desk, this does not necessarily imply that the user is interested in the subject "desk" or "dining table." Instead, the appearance of those common pieces of furniture may simply be incidental to the digital photographs.

[0050] In some embodiments, the topic identification engine 132 and / or the object frequency engine 128 may employ various numerical statistics, such as term frequency inverse document frequency or "TF-IDF", to determine whether a detected object should influence topic identification and / or to what extent a detected object should influence topic identification. For example, in some embodiments, TD-IDF may be used by, for example, the object frequency engine 128 to determine to what extent a particular detected object (which may be used as a "term" for "TF-IDF") should influence whether a particular topic is identified as potentially relevant to a user.

[0051] The knowledge graph engine 136 can be configured to provide access to the knowledge graph 138. In various embodiments, the knowledge graph 138 may include nodes representing known entities (and in some cases entity attributes) and edges connecting nodes and representing relationships between entities. For example, a "banana" node can be connected (e.g., as a child node) to a "fruit" node, which in turn can be connected (e.g., as a child node) to a "farm product" and / or "food" node. As another example, a restaurant called a "Hypothetical Café" can be represented by a node that also includes attributes such as its address, the type of food served, hours, contact information, etc. In some embodiments, the "Hypothetical Café" node can be connected to one or more other nodes, such as a "restaurant" node, a "business" node, a node representing the city and / or state in which the restaurant is located, etc., via an edge (e.g., representing a child-parent relationship).

[0052] In various embodiments, the topic identification engine 132 may utilize one or more of the signals previously mentioned in conjunction with information obtained from the knowledge graph engine 136 to identify one or more topics of potential interest to the user. For example, assume that various objects are detected in a plurality of digital photographs captured and / or retrieved by the user. In various embodiments, the knowledge graph engine 136 may determine that those species are all classified into a hierarchy of a particular genus, and determine that the presence of a variety of different species evidences potential interest in the genus.

[0053] As another example, assume that a user watches multiple digital videos. Each of these digital videos can be represented in the knowledge graph 138 and can be connected to related entities (e.g., actors, producers, entertainers, etc.). In various embodiments, the knowledge graph engine 136 can utilize the various entity relationships in the knowledge graph 138 to predict one or more topics that the user is potentially interested in (e.g., movies and / or genres that the user may be interested in, places that the user may be interested in, etc.) based on the multiple digital videos that the user is watching.

[0054] As yet another example, suppose a user watches a football game on one night, then a baseball game on another night, and then takes digital photos at a basketball game on a third night. In various implementations, the topic identification engine 132 may determine that the user has a broad interest in, for example, professional sports, based on entities such as teams associated with each of these events or the events themselves. If all of the teams are from the same city, then in some implementations, the topic identification engine 132 may identify this city as a topic of potential interest to the user.

[0055] Figure 2 Depicts an example scenario in which techniques may be employed to detect a problem based on various signals 250 1-N To identify one or more topics that are potentially relevant to the user or that the user is potentially interested in. Figure 2 The first signal is 250 1 In the form of a digital image captured by a user using a visual sensor 107 of a client device 106. Such a digital image may be stored, for example, in a cloud-based storage 122 by an online storage engine 120. As part of the upload process, the digital image may be analyzed, for example, by an object recognition client 110 and / or an object recognition engine 124, to detect one or more objects. Figure 2 In the example of , a pair of egrets are detected in the first digital image. In some embodiments, this may be sufficient for the topic identification engine 132 to identify topics such as "birds", "waterfowl", "egrets" as potentially relevant to the user. However, in other embodiments, further confirmation may be required.

[0056] Assume that the user later watches a digital video about bald eagles. In various embodiments, this fact and / or aspect of the digital video can be used by, for example, the topic identification engine 132 as another signal 250 for identifying a topic that is potentially relevant to the user. N-1 For example, both an eagle and a heron are "birds," and thus "birds" may be identified as a subject of potential interest to the user. Additionally or alternatively, a further signal 250 in the form of a user performing a bird-related search ("bird house supplies") may be analyzed. N , to further confirm the user's interest in birds.

[0057] Figure 3 Another example scenario is depicted in which the techniques described herein may be employed to identify topics that are potentially relevant to a user. Figure 3 A digital photograph 350 is depicted, which is taken by a user, for example using a client device 106 in the form of a smartphone. In this example, the digital photograph 350 captures a kitchen, which includes a microwave 356. 1 、Oven 356 2, Dining table 356 3 and dining table 356 3 Top drone 356 4 For example, the four objects and their corresponding locations in the digital photo 350 may be detected by the object recognition client 110 and / or the object recognition engine 124. Such data may be used by the subject identification engine 132 to identify one or more subjects that are potentially relevant to the user.

[0058] In some implementations, the topic identification engine 132 may consider whether each object detected in a digital media file belongs to a particular topic or genus to determine whether the object is exceptional (and therefore should be used to identify potentially related topics) or is merely incidental to the digital media file. Figure 3 , the topic identification engine 132, for example, with the assistance of the knowledge graph engine 136 and / or the object frequency engine 128, may determine that there are three detected objects 356 1 ,356 2 and 356 3 are more commonly seen in kitchens. Therefore, the presence of these objects merely suggests that the digital photo 350 was captured in a kitchen, not that the user is potentially interested in these objects. However, the drone 356 4 With other detected objects 356 1-3 In addition, the drone is relatively prominently featured in the digital photo 350. It is in the oven 356 2 and dining table 356 3 The subject identification engine 132 may determine that the drone 356 is "in front of" the drone 356 (or in the foreground relative to the drone 356). It is also relatively close to the center area of ​​the digital photograph 350 and is in focus. 4 is evidence that the user is interested in a topic such as drones. This potential interest may be further confirmed by other signals, such as drone-related search queries from the user, drone-related videos the user has watched, and so on.

[0059] Figure 4 Still another scenario is depicted in which the techniques described herein can be used to identify one or more topics that are potentially relevant to the user that the user may not be aware of in this example. In this example, the user obtains a first digital photo 450 of the user's son, for example using the visual sensor 107 of the client device 106. 1As can be seen, the son is wearing a T-shirt with the symbol "Ω" painted on it. For example, this symbol may be a symbol used by a musician that the son likes. The symbol, and in some cases the son's identity, may be detected by, for example, the object recognition client 110 and / or the object recognition engine 124. This data may be used, for example, by the topic identification engine 132 with the help of the knowledge graph engine 136 to determine that the symbol corresponds to a particular musician.

[0060] Later, the user obtains another digital image 450 again depicting the son. 2 At this point, the son is wearing a different T-shirt that does not depict the same symbol. However, the son's computer screen is currently displaying the same symbol as can be detected as described above. The presence of the symbol "Ω" in multiple images of the user's son can be used, for example, by the topic identification engine 132 to determine that the theme of the musician associated with the symbol "Ω" is potentially relevant to the user. For example, when the son's birthday is approaching, the father may be proactively informed of the musician's upcoming tour dates.

[0061] Once topics have been identified as potentially relevant to a user, they may be used for a variety of different purposes. In some embodiments, topics potentially relevant to a user may be used to sort search results presented to the user and / or formulate alternative query suggestions presented to the user. Additionally or alternatively, in some embodiments, topics potentially relevant to a user may be used to disambiguate user input. Suppose a user asks a personal digital assistant executed on a particular client device 106 to call a contact named "John". There may be multiple "Johns" in the user's contact list. In various embodiments, one or more digital media files (e.g., photos, voicemails, videos, etc.) created and / or retrieved by a user may be analyzed as described herein to identify "John" that the user is most likely to mention as a topic potentially relevant to the user, rather than requesting disambiguation of input from the user. Additionally or alternatively, topics determined using the techniques described herein may be used to recommend media for user consumption.

[0062] In some implementations, topics determined using the techniques described herein can be used to proactively present content to a user. Figure 5, a client device 506 in the form of a smartwatch is being used to present content to a user proactively (i.e., without user request). In this example, the techniques described herein have been used to determine that the user is interested in birds from digital media content created and / or retrieved by the user. Accordingly, the user is presented with a first "card" or "sticker" that recommends nearby exhibitions to bird watchers and a second "card" or "sticker" that recommends nearby bird-related movies. In various embodiments, these cards / stickers may be selectable deep links that, if selected, launch an appropriate user interface that enables the user to obtain more information, make reservations, etc.

[0063] In various embodiments, a user's engagement with content presented using the techniques described herein can be used to confirm and / or refute a topic identified using the techniques described herein. Figure 5 This may suggest that although the techniques described herein suggest that the user is potentially interested in birds, the user is not actually interested in birds. Therefore, the topic identification engine 132 may, for example, remove or downgrade the topic "birds" from the user interest database 134.

[0064] Now refer to Figure 6 , describes an example method 600 for implementing selected aspects of the present disclosure. For convenience, the operations of the flowchart are described with reference to a system that performs the operations. Such a system may include various components of various computer systems. For example, some operations may be performed at the client device 106h, while other operations may be performed by one or more components of the cloud-based system 102. In addition, although the operations of method 600 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, or added.

[0065] At block 602, the system, for example with the aid of the online storage engine 120, may receive data indicating one or more digital media files from one or more client devices (e.g., 106) operated by a user. In some embodiments, such data may be the original digital media files themselves, such as digital media files that a user is uploading (consciously or automatically) for backup purposes. Additionally or alternatively, in some embodiments, this may include data indicating detected objects generated locally on the client device, such as dimensionality reduction embeddings, etc. These digital media files have been created and / or retrieved for consumption by the user.

[0066] In block 604, the system, e.g., with the aid of object recognition client 110 and / or object recognition engine 124, may identify one or more objects detected in each of the one or more digital media files based on the data indicative of the one or more digital media files. For example, object recognition engine 124 may perform various types of object recognition (described previously) on the digital media files to detect one or more represented objects.

[0067] In block 606, the system, for example with the aid of the object frequency engine 128, may determine the frequency of each of the one or more objects detected in block 604 in the expected library of digital media files (e.g., 130) provided by the user population. As described above, various statistics such as TF-IDF may be employed to determine how common the detected objects are, and therefore whether and / or to what extent they should be used to identify topics of potential relevance to the user. Highly common objects may actually be noise rather than useful signals of interest to the user.

[0068] In block 608, the system may select one or more of the objects that should be used to identify potential subjects of interest based on the corresponding frequencies determined in block 606. For example, common objects such as furniture, trees, etc. may not be used to identify subjects of interest to the user unless, for example, those common objects occur frequently in the user's digital media content and / or are prominently featured in the digital media files in the absence of other less common objects (e.g., the user takes a lot of pictures of trees in the absence of other objects).

[0069] In block 610, the system, such as through the topic identification engine 132, can identify one or more topics that are potentially relevant to the user based on the one or more objects selected at block 608. In block 612, the system can formulate a search query based on the one or more topics identified. For example, if the topics "birds" and "waterfowl" are identified from the user's digital photos, a search query such as "birds waterfowl" can be formulated and submitted to a search engine to obtain responsive content. In some embodiments, the responsive content can be ranked based on those topics and / or other topics identified using the techniques described herein.

[0070] In block 614, the system may actively cause information responsive to the formulated search query to be output at one or more of the client devices operated by the user. Figure 5Non-limiting examples of such active content are depicted in . Additional examples of active content include, but are not limited to, text messages (e.g., SMS or MMS), social media messages (e.g., on a user's feed), natural language output from a personal digital assistant (e.g., during an existing human-computer dialogue between a user and a personal digital assistant, etc. or otherwise), email messages, pop-up windows (e.g., Android toasts), restaurant recommendations (e.g., provided in a restaurant reservation and / or review application), digital media recommendations (e.g., presented as part of a digital media application), and the like.

[0071] Now refer to Figure 7 , depicts another example method 700 according to various embodiments, which may be implemented in whole or in part on a client device 106. For convenience, the operations of the flowchart are described with reference to a system that performs the operations. Such a system may include various computer systems, particularly various components of the client device 106, although one or more operations may additionally or alternatively be performed by one or more components of the cloud-based system 102. Furthermore, while the operations of the method 700 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, or added.

[0072] At box 702, the system can obtain one or more digital files created by a user using a client device. For example, in some embodiments, when a user takes a digital photo, data representing the digital photo can be stored in volatile and / or non-volatile memory. At box 704, the system, for example with the aid of the object recognition client 110, can apply the one or more digital media files as input to one or more trained machine learning models local to the client device. For example, the object recognition client 110 can apply the one or more digital media files in one or more trained machine learning models (e.g., CNN) stored in the model index 111 on the device. In various embodiments, the one or more trained machine learning models can generate an output indicating one or more objects detected in each of the one or more digital media files.

[0073] At block 706, the system (e.g., client device 106) may provide data indicating the indicated one or more objects to a remote computing system such as cloud-based system 102, without providing the one or more digital media files themselves. Thus, the data provided at block 706 may be different from the digital media files themselves, such as in the form of a vector of features extracted from the digital media files, a reduced dimensionality embedding generated based on the digital media files, etc. For example, instead of providing the original digital photo data to the cloud-based system 102, the client device 106 may provide output generated from one or more CNNs to the cloud-based system 102. For example, such output may include one or more object predictions and corresponding confidence metrics (e.g., 70% probability that the object is a cat, 10% probability that the object is a dog, 5% probability that the object is a monkey, etc.). Notably, providing such data to the cloud-based system 102, rather than exposing the original data including the user's digital media files to the cloud-based system 102, may protect the user's privacy to the extent that the digital media files contain information that the user considers private or sensitive.

[0074] At block 708, the system may receive information associated with the indicated one or more objects from the same remote computing system or a different remote computing system. For example, the client device 106 may receive information associated with the indicated one or more objects from the cloud-based system 102 or another remote computing system such as a search engine that returns search results in response to a request made in the client device 106. Figure 6 The search query formulated at block 612 of ) receives information associated with one or more objects and / or topics inferred from one or more topics. In various embodiments, the objects themselves (e.g., their classes or types) can form "topics" that are considered potentially relevant to the user by, for example, the topic identification engine 132.

[0075] In block 710, the system may proactively output information associated with the indicated one or more objects to the user, e.g., at the same client device used to create the digital media file or at a different client device operated by the user (such as another client device of a coordinated "ecosystem" of client devices operated by the user). For example, assume that a user captures a photo using her smartphone. After identifying one or more topics of potential interest to the user from the photo using the techniques described herein, output related to those topics of potential interest may be presented at another client device of the user, e.g., as a card or sticker on a smart watch or smart TV. Some examples of proactive output are described in

[0066] FIG. 710. Figure 5 is provided in, and about Figure 6 614 is described below.

[0076] As another example of active output, assume that the user has previously posted digital photos of many meals that the user has prepared and / or consumed on social media. The technology described in this article can be used to identify the user's potential interest in cooking from the digital photos posted to social media. Now, assume that the user later participates in a voice-based human-computer dialogue with an automated assistant, for example, through a stand-alone interactive speaker. It is further assumed that there are third-party applications or "agents" related to cooking that the user can participate in via the automated assistant but the user does not know. Based on the user's perceived interest in cooking, the automated assistant may actively introduce the user to the third-party agent. For example, the automated assistant can provide audible natural language output, such as "I see that you have posted several pictures of meals on your social media account. This is a friend of mine who is happy to recommend various recipes." In some cases, the third-party agent may then interject: "Hi, let me know what ingredients you have or how you are feeling, and I will recommend recipes. "

[0077] Figure 8 810. The computer system 810 generally includes at least one processor 814 that communicates with a number of peripheral devices via a bus subsystem 812. These peripheral devices may include a storage subsystem 824 (e.g., a memory subsystem 825 and a file storage subsystem 826), a user interface output device 820, a user interface input device 822, and a network interface subsystem 816. The input and output devices allow a user to interact with the computer device 810. The network interface subsystem 816 provides an interface to an external network and is coupled to corresponding interface devices in other computer systems.

[0078] The user interface input device 822 may include a keyboard, a pointing device (such as a mouse, trackball, touch pad, or graphics tablet), a scanner, a touch screen incorporated into a display, an audio input device (such as a voice recognition system, a microphone), and / or other types of input devices. In general, the use of the term "input device" is intended to include all possible types of devices and ways of entering information into the computer system 810 or onto a communication network.

[0079] The user interface output device 820 may include a display subsystem, a printer, a fax machine, or a non-visual display, such as an audio output device. The display subsystem may include a cathode ray tube (CRT), a flat panel device (such as a liquid crystal display (LCD)), a projection device, or some other mechanism for creating a visible image. The display subsystem may also provide a non-visual display, such as through an audio output device. In general, the use of the term "output device" is intended to include all possible types of devices and methods for outputting information from the computer system 810 to a user or another machine or computer system.

[0080] The storage subsystem 824 stores programming and data structures that provide some or all of the functionality of the modules described herein. For example, the storage subsystem 824 may include logic for executing the methods 600, 700 and / or implementing one or more of the object recommendation client 110, the online storage engine 120, the object recognition engine 124, the object frequency engine 128, the topic identification engine 132, and / or the knowledge graph engine 136.

[0081] These software modules are usually executed by processor 814 alone or in combination with other processors. The memory 825 for the storage subsystem can include multiple memories, including a main random access memory (RAM) 830 for storing instructions and data during program execution and a read-only memory (ROM) 832 storing fixed instructions. The file storage subsystem 824 can provide permanent storage for program and data files, and can include a hard drive, a floppy disk drive and associated removable media, a CD-ROM drive, an optical drive or a removable media cartridge. The modules implementing the functions of certain embodiments can be stored in the storage subsystem 824 or other machines accessible by (multiple) processors 814 by the file storage subsystem 826.

[0082] The bus subsystem 812 provides a mechanism for allowing the various components and subsystems of the computer system 810 to communicate with each other as intended. Although the bus subsystem 812 is shown schematically as a single bus, alternative implementations of the bus subsystem may use multiple busses.

[0083] Computer system 810 can be of various types, including a workstation, server, computing cluster, blade server, server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, the Figure 8 The description of the computer system 810 depicted in FIG. 8 is merely a specific example for illustrating some embodiments. Many other configurations of the computer system 810 may have more Figure 8 The computer system may have more or fewer components than those depicted.

[0084] In the case where the system described in this article collects personal information about a user or may use personal information, the user may be provided with the following opportunities: control whether a program or feature collects user information (e.g., information about the user's social network, social actions or activities, occupation, user's preferences, or the user's current geographic location), or control whether and / or how to receive content that may be more relevant to the user from a content server. Similarly, certain data may be processed in one or more ways before being stored or used so that personally identifiable information is removed. For example, the user's identity may be processed so that the user's personally identifiable information cannot be determined, or the user's geographic location may be generalized (such as to a city, ZIP code, or state level) in the case of obtaining geographic location information so that the user's specific geographic location cannot be determined. Therefore, the user can control how information about the user is collected and / or used information.

[0085] Although various embodiments have been described and illustrated herein, various other means and / or structures for performing functions and / or obtaining results and / or one or more of the advantages described herein may be utilized, and each of such changes and / or modifications is considered to be within the scope of the embodiments described herein. More generally, all parameters, sizes, materials, and configurations described herein are meant to be exemplary, and the actual parameters, sizes, materials, and / or configurations depend on one or more specific applications in which the present teachings are used. Those skilled in the art will recognize or determine many equivalents of the specific embodiments described herein using only routine experiments. Therefore, it is to be understood that the aforementioned embodiments are presented only by way of example, and within the scope of the attached claims and their equivalents, the embodiments may be practiced in a manner different from that specifically described and claimed. The embodiments of the present disclosure relate to each individual feature, system, article, material, kit, and / or method described herein. In addition, if these features, systems, articles, materials, kits, and / or methods are not mutually contradictory, any combination of two or more such features, systems, articles, materials, kits, and / or methods is included within the scope of the present disclosure.

Claims

1. A computer-implemented method for predicting potentially relevant topics, comprising: receiving data indicative of one or more digital images from one or more client devices operated by a user; identifying one or more objects detected in each of the one or more digital images by performing, by one or more of the processors, image recognition processing on data indicative of the one or more digital images; determining a frequency at which each of the one or more objects occurs in a corpus of digital images provided by a community of users; identifying one or more topics potentially relevant to the user based on the frequency with which the one or more objects appear in the corpus, wherein the frequency with which a given object appears in the corpus is inversely proportional to the likelihood that a topic will be identified from the object; formulating a search query based on the identified one or more topics; and Information responsive to the formulated search query is actively caused to be output at one or more of the client devices operated by the user.

2. The computer-implemented method for predicting potentially relevant topics of claim 1, wherein: The data indicative of the one or more digital images comprises one or more feature vectors or reduced dimensionality embeddings extracted from the one or more digital images.

3. The computer-implemented method for predicting potentially relevant topics of claim 1, wherein: The one or more topics potentially relevant to the user are further identified based on the location of the one or more objects in the one or more digital images.

4. The computer-implemented method for predicting potentially relevant topics of claim 1, wherein: The one or more topics potentially relevant to the user are further identified based on a measure of focus of the one or more objects in the one or more digital images.

5. The computer-implemented method for predicting potentially relevant topics of claim 1, wherein: The one or more topics potentially relevant to the user are further identified based on whether the user created or retrieved the one or more digital images received from the one or more client devices.

6. The computer-implemented method for predicting potentially relevant topics of claim 1, wherein: Determining a frequency at which each of the one or more objects occurs in the corpus of digital images provided by the user community includes applying term frequency-inverse document frequency ("TF-IDF").

7. A system for predicting potentially relevant topics, the system comprising one or more processors and a memory operably coupled to the one or more processors, wherein: The memory stores instructions which, when executed by the one or more processors, cause the one or more processors to perform the method according to any one of claims 1-6.

8. At least one non-transitory computer-readable medium comprising instructions which, when executed by one or more processors, cause the one or more processors to perform the method according to any one of claims 1 to 6.

Images