Generative machine learning model for identifying optimal user-generated representation images

Abstract

One or more visual descriptors for a particular point of interest (POI) are extracted from a user generated content (UGC) corpus, wherein the UGC corpus includes a plurality of textual content items describing the particular POI. Based on the one or more visual descriptors, a machine learning generative visual model is used to generate a query image depicting the particular POI. A visual similarity between the query image and each of a plurality of candidate images of the visual UGC corpus depicting the particular POI is determined. A representation image is selected from the plurality of candidate images to represent the particular POI based on a visual similarity between the representation image and the query image.

Description

[0001] Priority declaration

[0002] This application claims the benefit of and priority to U.S. patent application No. 18 / 393,110, filed on December 21, 2023, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] The present disclosure generally relates to identifying optimal representation images. More specifically, the present disclosure relates to utilizing a generative machine-learned model to identify optimal user-generated images to represent specific points of interest (POIs). Background Art

[0004] Application-based services are increasingly common in the modern world. Application-based services typically provide services to users via applications executed by the user's device. Examples of such services include map services, aggregation services (e.g., for user reviews, etc.), visual search services, etc. One benefit provided by such services is that it enables the exchange of user-generated content (UGC). For example, a map service may maintain "profiles" of various points of interest (POIs) (e.g., restaurants, landmarks, geographic areas, businesses, etc.).

[0005] The profile of a POI is used to aggregate and categorize UGC submitted for the POI. In addition, for many application-based services, the profile of a POI typically includes "overview" information, which is first presented to the user to provide a high-level overview of the POI. This overview information typically includes information describing the POI (e.g., business hours of the business, menus of restaurants, highlighted sentences in popular user reviews, etc.) and a representation image that serves as a visual "overview" of the POI. For example, the representation image included in the map service profile of a restaurant may be a panoramic depiction of the interior of the restaurant, a depiction of the restaurant's front entrance, an image of a signature dish served by the restaurant, etc. Summary of the invention

[0006] Aspects and advantages of embodiments of the present disclosure will be set forth in part in the following description, or may be learned from the description, or may be learned through practice of the embodiments.

[0007] An example aspect of the present disclosure relates to a computer-implemented method. The method includes: extracting one or more visual descriptors for a specific point of interest (POI) from a user-generated content (UGC) corpus by a computing system including one or more processor devices, wherein the UGC corpus includes multiple text content items describing the specific POI. The method includes: based on the one or more visual descriptors, the computing system uses a machine learning generative visual model to generate a query image depicting the specific POI. The method includes: determining, by the computing system, a visual similarity between the query image and each of a plurality of candidate images in the visual UGC corpus depicting the specific POI. The method includes: selecting, by the computing system, a representative image from a plurality of candidate images to represent the specific POI based on the visual similarity between the representative image and the query image.

[0008] Another example aspect of the present disclosure relates to a computing system. The computing system includes one or more processor devices. The computing system includes: one or more tangible non-transitory computer-readable media, the one or more tangible non-transitory computer-readable media storing computer-readable instructions, which when executed by one or more processor devices cause the one or more processor devices to perform operations. The operation includes: extracting one or more visual descriptors for a specific POI from a UGC corpus, wherein the UGC corpus includes multiple text content items describing the specific POI. The operation includes: based on the one or more visual descriptors, using a machine learning generative visual model to generate a query image depicting the specific POI. The operation includes: determining the visual similarity between the query image and each of the multiple candidate images of the visual UGC corpus depicting the specific POI. The operation includes: selecting the representative image to represent the specific POI based on the visual similarity between the representative image in the multiple images and the query image.

[0009] Another example aspect of the present disclosure relates to one or more tangible non-transitory computer-readable media storing computer-readable instructions that, when executed by one or more processor devices, cause the one or more processor devices to perform operations. The operations include: extracting one or more visual descriptors for a specific POI from a UGC corpus, wherein the UGC corpus includes multiple text content items describing the specific POI. The operations include: based on the one or more visual descriptors, using a machine learning generative visual model to generate an embedding, from which a query image depicting the specific POI can be generated. The operations include: determining the similarity between the embedding and each of multiple embeddings of multiple candidate images that respectively represent the visual UGC corpus depicting the specific POI. The operations include: selecting a representative image from a plurality of images to represent the specific POI based on the similarity between the embedding and each of the multiple embeddings.

[0010] Other aspects of the disclosure relate to various systems, apparatuses, non-transitory computer-readable media, user interfaces, and electronic devices.

[0011] These and other features, aspects and advantages of various embodiments of the present disclosure will be better understood with reference to the following description and appended claims.The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate example embodiments of the present disclosure and, together with the description, serve to explain the relevant principles. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] A detailed discussion of the embodiments for those skilled in the art is set forth in this specification with reference to the accompanying drawings, in which:

[0013] Figure 1 is an overview block diagram of a computing system for a generative machine learning model to identify optimal user-generated images to represent a particular POI according to some implementations of the present disclosure.

[0014] Figure 2 is a block diagram of a computing system including a machine learning module for selecting a representative image for a POI using a generative machine learning model according to some implementations of the present disclosure.

[0015] Figure 3 is a block diagram of a computing system that identifies representative images from a corpus of user-generated content based on visual similarity between the image and a query image, according to some implementations of the present disclosure.

[0016] Figure 4is a block diagram of a computing system that modifies a POI profile to replace an existing representation image with a newly selected representation image, according to some implementations of the present disclosure.

[0017] Figure 5 An example profile for a particular POI within a map application associated with a map service is shown in accordance with some implementations of the present disclosure.

[0018] Figure 6 Depicted is a flowchart of an example method for performing representation image selection according to an example embodiment of the present disclosure.

[0019] Fig. 7A Depicted is a block diagram of an example computing system that performs selection of a representative image corresponding to a POI profile according to an example embodiment of the present disclosure.

[0020] Figure 7B A block diagram of an example computing device that trains a model for representing image selection according to an example embodiment of the present disclosure is depicted. The computing device may be a user computing device or a server computing device.

[0021] Figure 7C A block diagram of an example computing device utilizing a machine learning model for representative image selection is depicted according to an example embodiment of the present disclosure.

[0022] Reference numerals repeated across multiple drawings are intended to identify like features in the various implementations. DETAILED DESCRIPTION

[0023] In general, the present disclosure relates to identifying optimal representation images. More specifically, the present disclosure relates to utilizing a generative machine learning model to identify optimal user-generated images to represent specific points of interest (POIs). Application-based services are becoming increasingly common in the modern world. Application-based services typically provide services to users via applications executed by the user's device. Examples of such services include map services, aggregation services (e.g., for user reviews, etc.), visual search services, etc. One benefit provided by such services is that it enables the exchange of user-generated content (UGC). For example, a map service can maintain "profiles" of various points of interest (POIs) (e.g., restaurants, landmarks, geographic areas, businesses, etc.).

[0024] The profile of a POI is used to aggregate and categorize UGC submitted for the POI. In addition, for many application-based services, the profile of a POI typically includes "overview" information, which is first presented to the user to provide a high-level overview of the POI. This overview information typically includes information describing the POI (e.g., business hours of the business, menus of restaurants, highlighted sentences in popular user reviews, etc.) and a representation image that serves as a visual "overview" of the POI. For example, the representation image included in the map service profile of a restaurant may be a panoramic depiction of the interior of the restaurant, a depiction of the restaurant's front entrance, an image of a signature dish served by the restaurant, etc.

[0025] The representative image is typically selected from UGC images submitted in association with a particular POI. For example, users of a mapping service often capture images while dining at a restaurant and upload these images to the mapping service along with a review of the restaurant. Conventional approaches select representative images for a POI based on several signals, such as interaction metrics for the UGC image, feedback from representatives of the POI, image quality metrics, recency, etc.

[0026] However, although conventional approaches are successful in selecting "hot" images with frequent interactions, such images often serve as poor representation images for POIs. In some instances, "hot" images may instead incorrectly represent specific POIs. For example, suppose the most popular UGC image of a restaurant depicts a close-up of a signature menu item. Although popular, the image does not give an indication of the aesthetics, atmosphere, appearance, etc. of the restaurant, and therefore can serve as a poor representation image for a POI. In addition, conventional approaches often exhibit poor performance in terms of the temporal and / or contextual relevance of images. Specifically, conventional approaches are generally unable to change the representation image of a POI based on planned events, seasonal weather changes, etc. For example, suppose a POI is a national park famous for its bright autumn leaves. Conventional approaches are unlikely to change existing representation images to images captured in previous autumns to highlight such leaves.

[0027] Therefore, implementations of the present disclosure propose a generative machine learning model for identifying optimal user-generated images to represent a particular POI. More specifically, a computing system (e.g., a system associated with an application-based service) may maintain a UGC corpus submitted by users for a particular POI. The UGC corpus may include text content items describing the POI (e.g., reviews for the POI, frequently asked questions / answers for the POI, an overview or overview of the POI, POI-specific and / or POI-submitted information, etc.). The UGC corpus may also include multiple images depicting various parts or features of the POI. For example, if the POI is a restaurant, the UGC corpus may include images of the restaurant's exterior, interior, various menu items, special events occurring at the restaurant, and the like.

[0028] The computing system may extract visual descriptors for a particular POI from the UGC corpus. As described herein, a "visual descriptor" may be a word, phrase, sentence, etc. that describes the visual characteristics of the POI, the goods or services provided by the POI, the activities performed at the POI, the people associated with the POI, etc. For example, if the POI is a national park, visual descriptors such as "hike", "fall leaves", "lots of flowers in summer", "beautiful view from the mountain", etc. may be extracted.

[0029] Based on the visual descriptors, the computing system can use the machine learning generative visual model to generate a query image depicting a specific POI. More specifically, the computing system can utilize the machine learning generative visual model to process the visual descriptors or intermediate representations of the visual descriptors to generate an image depicting the POI based on the visual descriptors. For example, based on the visual descriptors "fall leaves" and "beautiful view from the mountain" for a national park, the generated image can depict a mountain completely covered with autumn leaves.

[0030] In this way, the computing system can generate an image that can serve as an "ideal" image for identifying visually similar images within a UGC image corpus. More specifically, the computing system can determine the visual similarity between a query image (e.g., an image generated using a generative vision model) and images included in the UGC corpus. By identifying the UGC image that is most visually similar to the "ideal" query image, the computing system can identify the best representation image for the POI.

[0031] Following the previous example, one of the images included in the UGC corpus might be an image uploaded by a user several years ago depicting a national park during the peak fall foliage season. Conventional image selection techniques are unlikely to select such an image due to the length of time that has elapsed since the image was captured. However, implementations of the present disclosure may select the image due to the visual similarity between the image and an "ideal" query image. In this way, a computing system may more accurately and efficiently identify a representative image for a POI based on semantic context understanding of the POI extracted from text content items and other UGC.

[0032] Implementations of the present disclosure provide several technical effects and benefits. As an example technical effect and benefit, the implementations described herein more accurately and efficiently identify query images for POIs. In addition, the implementations described herein enable more dynamic selection of hero images based on temporal context. For example, if the owner of a venue provides information indicating that an electronic music artist is scheduled to perform at the venue, the implementations described herein can be used to temporarily select a representative image (e.g., from a previous electronic music artist, etc.) corresponding to the upcoming event.

[0033] Referring now to the drawings, example implementations of the present disclosure will be discussed in greater detail.

[0034] Figure 1 1 is an overview block diagram 100 of a computing system for a generative machine learning model to identify an optimal user-generated image to represent a specific POI according to some implementations of the present disclosure. Specifically, a computing system 102 (e.g., a physical computing device, a virtualized computing device, etc.) may include a service handler 104 that provides or implements an application-based service or an application-based service or a component corresponding to an application. For example, the service handler 104 may provide or implement a map service, a navigation service, a visual search service, etc.

[0035] The service handler 104 may include a POI profile manager 106. The POI profile manager 106 may create, modify, and maintain profile information for various POIs. Specifically, some application-based services (e.g., navigation services) maintain profile information for POIs that can be discovered within application-based services. For example, POIs such as restaurants, stores, and other businesses may be discovered through application-based services such as navigation services, visual search services, and the like. The profile may include information describing various characteristics of the POI, such as business hours, phone numbers, associated websites, and the like. The profile may also store UGC submitted by users who have visited the POI. For example, the profile of a restaurant in a navigation or map application may include user-submitted reviews, images of meals served to users, and the like.

[0036] POI profile manager 106 may include text UGC repository 108. Text UGC repository 108 may store text content items 110A-110N (generally, text content items 110) submitted by users for POI. Text content items 110 may include text content of any type or manner, including comments, questions / answers, overviews, extracts from other text sources, quotations, etc. In some implementations, text content items 110 may include information extracted from websites, social media profiles, application profiles, etc. associated with specific POIs. In some implementations, text content items 110 may be the output of text conversion technology applied to some other forms of user-generated content. For example, text content items may be the output of speech recognition technology applied to user-generated audio content. For another example, text content items may be the output generated by processing UGC images using a machine learning semantic image recognition model trained to generate a text description of an input image.

[0037] The POI profile manager may include a visual UGC repository 112. The visual UGC repository 112 may include any visual or audiovisual content (e.g., images, animations, videos, mixed reality (MR) information, etc.) submitted by users for a POI. In particular, the visual UGC repository 112 may include UGC images 114A-114N (collectively, UGC images 114). The UGC images 114 may depict a POI, a portion of a POI, or a subject related to a POI (e.g., a menu at a restaurant, a logo of a business, a service provider providing a service, etc.).

[0038] The POI profile manager 106 may include a POI profile information repository 116. The POI profile information repository 116 stores information displayed within the profile of the POI, and / or metadata associated with the POI. For example, assume that the computing system 102 is associated with a map or navigation service. The POI profile information repository 116 may include profile information for profiles of businesses that can be discovered using the navigation service. The profile information stored in the repository 116 may include the hours that the business is open, the address of the business, the coordinates of the business, associated websites, etc. In some implementations, the POI profile information repository 116 may include contextual information specific to the POI and / or the type of POI.

[0039] As previously described, the service implemented by the service handler can maintain representation images to represent various POIs. For users browsing the profile of the POI, the representation image can serve as a "visual overview" of the POI (or the goods or services associated with the POI). The representation image is usually displayed in a prominent position within the profile of the POI to attract the user's attention. For example, the representation image for an amusement park may depict the most famous rides in the park. For another example, the representation image for a restaurant may depict the restaurant's logo, the popular dishes served by the restaurant, the interior of the restaurant, etc.

[0040] The computing system 102 may include a representation image identifier 118 to optimally select a representation image for a POI. The representation image identifier 118 may identify the best representation image from the existing UGC images 114 of the visual UGC repository 112. To this end, the representation image identifier 118 may include a visual descriptor extractor 120. The visual descriptor extractor 120 may extract visual descriptors from the text content items 110 included in the text UGC repository 108.

[0041] Visual descriptor extractor 120 may include text UGC corpus 122. Text UGC corpus may include selected text content items 124A-124D (collectively, selected text content items 124). In some implementations, selected text content items 124 may include each of text content items 110 from text UGC repository 108. Text content items 124

[0042] Alternatively, in some implementations, the visual descriptor extractor 120 may filter one or more of the text content items 110 from the selected text content items 124. In some implementations, the text content items 110 may be filtered based on user feedback submitted for the text content items 110. For example, the POI profile information repository 116 may include feedback information submitted by a user for the text content items 110. If the feedback information for the text content item 110A indicates negative feedback or a certain feedback metric, the visual descriptor extractor 120 may filter the text content item 110A, or may otherwise avoid selecting the text content item 110A to be included in the selected content items 124. Additionally or alternatively, in some implementations, the visual descriptor extractor 120 may filter the text content items 110 based on other criteria, such as inappropriate or insensitive language, predicted marketing content, predicted malicious content, predicted inaccuracy, etc.

[0043] The visual descriptor extractor 120 may extract visual descriptors from the selected text content items 124 of the text UGC corpus 122. More specifically, the visual descriptor extractor 120 may generate visual descriptor information 126. The visual descriptor information 126 may include visual descriptors 128A-128N (collectively, visual descriptors 128). As described herein, a "visual descriptor" may be a word, sentence, phrase, etc. that describes the visual characteristics of a POI and / or a service or commodity associated with the POI. For example, the visual descriptors 128 extracted for a candy store POI may include words such as "colorful", "fun", etc., while the visual descriptors 128 for a national park may include words such as "scenic" or "breathtaking".

[0044] In some implementations, the visual descriptor extractor 120 may use conventional extraction techniques to extract the visual descriptors 128. For example, the visual descriptor extractor 120 may perform a search process to search for known visual descriptors. Additionally or alternatively, in some implementations, the visual descriptor extractor 120 may use machine learning techniques to extract the visual descriptors 128. For example, the visual descriptor extractor 120 may process the text UGC corpus 122 and a prompt describing instructions to extract the visual descriptors 126 from the selected text content item 124.

[0045] The representation image recognizer 118 may include a machine learning module 130. The machine learning module 130 may train, store, instantiate, optimize, and otherwise implement various machine learning models 132. The machine learning module 130 may include model information 134. The model information 134 may include hints for implementing the machine learning model 132, versioning information, update information, file paths, and the like.

[0046] In some implementations, the machine learning model 132 may include at least a portion of a machine learning language model, such as a large language model (LLM). For example, the machine learning model 132 may include a trained encoder portion of the LLM that is trained to process text content to generate an intermediate representation of the text content. In some implementations, the trained encoder portion of the LLM may be a portion or sub-model of a multimodal model. For example, a multimodal model may include a trained encoder portion of the LLM and similar encoder portions for visual tasks, audio tasks, and the like. In some implementations, the trained encoder portion of the LLM may be trained so that the intermediate representation output by the encoder portion can be processed by the decoder portion of the machine learning computer vision model.

[0047] More specifically, the machine learning model 132 may include a machine learning computer vision model trained to perform a generative image task. In other words, the machine learning computer vision model may be trained to generate a requested image based on a description of the requested image. In some implementations, the trained encoder portion of the LLM may be trained in a manner such that the output of the trained encoder portion is compatible with the machine learning computer vision model. For example, assume that the machine learning computer vision model has an encoder-decoder architecture. Additionally or alternatively, in some implementations, the machine learning computer vision model may be an end-to-end multimodal model (e.g., a language vision model, etc.) trained to process textual content to generate images based on the textual content. Therefore, it should be noted that the machine learning model 132 may include any type of model organized in any manner sufficient to generate an image based on a textual description of the image.

[0048] Additionally or alternatively, in some implementations, the machine learning model 132 may include a model trained to generate intermediate representations of other types of information (such as audio information) or to generate text summaries of such types of information. For example, the machine learning model 132 may include an audio encoder model trained to perform speech recognition tasks. For another example, the machine learning model 132 may include a machine learning computer vision model trained to generate a text summary of the content depicted by an image.

[0049] Machine learning module 130 may include contextual information 136. Contextual information 136 may describe various characteristics of the POI, the services or goods provided by the POI, and / or events or situations related to the POI. For example, contextual information 136 may identify the type of goods or services provided by the POI. For another example, if the POI is a landmark, contextual information 136 may include historical information related to the POI.

[0050] In some implementations, the contextual information 136 may include time information. The time information may be time-sensitive contextual information, such as seasonal changes, current or predicted weather, current or predicted traffic conditions, current or planned events associated with the POI, recent or planned modifications to the POI, and the like. For example, a business entity associated with an event space POI may upload information to the profile of the POI indicating that the POI will host an electronic music festival in the near future. The contextual information 136 may include this information, or may otherwise indicate a planned electronic music festival. In this way, a representation image that is most relevant to a planned event or a predicted event may be selected.

[0051] In some implementations, contextual information 136 may be or otherwise include prompts for machine learning model 132. For example, contextual information 136 may include prompts that instruct machine learning model 132 to generate an image that depicts a POI and includes each of the visual descriptors 128. As another example, contextual information 136 may include prompts that instruct the machine learning model to generate an image based on some of the visual descriptors 128. In some implementations, contextual information 136 may include prompts based on contextual information 136 itself. For example, contextual information 136 may include information indicating an upcoming season change from a current season to a subsequent season. The prompts may instruct machine learning model 132 to depict POIs during the subsequent season rather than during the current season. In this way, the prompts processed together with the visual descriptors 128 may improve the accuracy of machine learning model 132.

[0052] The representation image recognizer 118 may utilize the machine learning model 132 to process the model input 138 to obtain the query image 140. The model input 138 may include the visual descriptors 128, the context information 136, etc. The query image 140 may depict an “ideal” depiction of the POI based on the visual descriptors 128. For example, if the visual descriptors include a depiction of autumn leaves, the query image 140 may depict autumn leaves.

[0053] The representation image identifier 118 may include a visual similarity identifier 142. The visual similarity identifier 142 may include the query image 140 and a visual UGC corpus 144. Similar to the textual UGC corpus 122, the visual UGC corpus 144 may include some or all of the UGC images 114 of the visual UGC repository 112. Further, the visual UGC corpus 144 may be filtered in the same manner as described with respect to the textual UGC corpus 122.

[0054] Visual similarity identifier 142 may determine the visual similarity between query image 140 and each image of visual UGC corpus 144. Based on the visual similarity, visual similarity identifier 142 may select representative image 146 from the images included in visual UGC corpus 144. Representative image identifier 118 may identify the best image to serve as representative image 146 by selecting the UGC image 114 that is most visually similar to the “ideal” depiction of the POI provided by query image 140.

[0055] It should be noted that in some implementations, the query image 140 may not be generated using the machine learning generative visual model 214. Instead, the machine learning generative visual model 214 may instead be used to generate an embedding, from which the query image 140 can be generated. More specifically, some generative visual models are structured to include an encoder portion and a decoder portion. The encoder portion can process some input to generate an intermediate representation of the input, such as an embedding. The decoder portion can process the embedding to generate the query image 140. However, performing the generative task using the decoder portion can be computationally expensive. Therefore, the encoder portion of the machine learning generative visual model 214 can process the model input 138 to generate an embedding instead of generating the query image 140. Once the embedding is generated, the similarity between the embedding and other embeddings generated from the UGC image 114 can be determined. Reference Figure 3 The evaluation of embeddings is discussed in more detail.

[0056] In some implementations, the representation image identifier 118 may include an evaluation event occurrence detector 148. The evaluation event occurrence detector 148 may detect the occurrence of a representation image evaluation event. A "representation image evaluation event" may be an event that provides a "reason" for evaluating or re-evaluating the current representation image for a POI profile. In some implementations, the representation image evaluation event may be a regularly scheduled event that is repeated iteratively. For example, the evaluation event occurrence detector 148 may detect a representation image evaluation event by detecting the passage of a predetermined time period after detecting a prior representation image evaluation event. In other words, each time a representation image is evaluated or re-evaluated, the evaluation event occurrence detector 148 may initialize a timer that, once expired, may cause a subsequent evaluation event.

[0057] In some implementations, the evaluation event occurrence detector 148 can detect the occurrence of a representative image evaluation event by receiving information indicating a request to evaluate a representative image. For example, the computing system 102 or some other computing entity associated with the computing system 102 can provide the request to the computing system. As another example, the computing system 102 can receive the request from a user (e.g., a user associated with the POI, such as an employee, etc.). In some implementations, the evaluation event occurrence detector 148 can detect the occurrence of a representative image evaluation event by determining that the seasonal conditions depicted in the representative image currently used to represent the particular POI are different from the current seasonal conditions.

[0058] In some implementations, the evaluation event occurrence detector 148 may detect the occurrence of a representation image evaluation event based on user feedback for the current representation image. For example, the evaluation event occurrence detector 148 may obtain user feedback information indicating the degree of representation accuracy of the representation image currently used to represent a specific POI. The evaluation event occurrence detector 148 may determine that the degree of representation accuracy of the representation image currently used to represent the specific POI is less than a threshold degree of representation accuracy.

[0059] Figure 2 is a block diagram 200 of a computing system including a machine learning module to select a representative image for a POI using a generative machine learning model, according to some implementations of the present disclosure. Figure 2 Will combine Figure 1 Specifically, the computing system 102 (e.g., Figure 1 The computing system 102 of the present invention may include a representation image recognizer 118. The representation image recognizer 118 may include a machine learning module 130. The machine learning module 130 may include a machine learning model 132, model information 134, context information 136, and a model input 138.

[0060] The computing system 102 can obtain a set of model inputs 138. The set of model inputs 138 can include visual descriptors 128. As shown, the visual descriptors 128 can be visual descriptors extracted from text content items (e.g., comments) submitted by users for the POI. According to the depicted example, the POI can be a ski resort that operates as a mountain biking park during the off-season. For example, the visual descriptors include descriptors such as "downhill mountain biking", "summer", "winter", "...ski mountain", "Santa Fe bikes", "K3 snowboards", etc. Furthermore, the visual descriptors 128 can determine the generated image.

[0061] In some implementations, model input 138 may include context information 136. In some implementations, context information 136 may include prompts 202. Prompts 202 may be or include one or more prompts for the machine learning model. In some implementations, prompts 202 may be textual content. Alternatively, in some implementations, prompts 202 may be encoded prior to processing with machine learning model 132. For example, prompts 202 may be encoded as an intermediate representation prior to processing.

[0062] In some implementations, the context information 136 may include time information 204. The time information 204 may include time-sensitive context information associated with a recent occurrence, a current occurrence, or a predicted occurrence. For example, the time information 204 may describe an event that is scheduled to occur at the POI in the near future. As another example, the time information 204 may include current weather conditions and / or predicted weather conditions. As another example, the time information 204 may include seasonal information (e.g., information indicating the beginning of a new season, etc.). In some implementations, the time information 204 may describe recent or planned modifications to the POI. For example, the time information 204 may indicate that the POI has just been renovated. As another example, the time information 204 may indicate that the POI has moved to a different building. As another example, the time information 204 may indicate that modifications have been made to materials associated with the POI (e.g., a new menu for a restaurant, new business hours for a business, new services provided by a service provider, etc.).

[0063] Additionally or alternatively, in some implementations, the temporal information 204 may describe a recent or planned modification to another POI associated with the particular POI. For example, assume that the POI is Yosemite National Park, which includes several different hotels and resorts located in the valley of the park. If one of the hotels located within Yosemite National Park is undergoing exterior renovations, the temporal information 204 may indicate this.

[0064] Additionally or alternatively, in some implementations, the contextual information 136 may include POI-specific contextual information 206. The POI-specific contextual information 206 may include contextual information specific to a particular POI and / or a particular type of POI. For example, assume that the POI is a fast food restaurant with regional variations in menu offerings. The POI-specific contextual information 206 may include information describing the characteristics of a particular restaurant (e.g., menu offerings specific to the restaurant), and / or may include information describing the characteristics of fast food restaurants in general (e.g., information related to drive-through ordering systems, etc.). For another example, if the POI is a national park, the POI-specific contextual information 206 may include or otherwise indicate a park map, rules within the park, information for obtaining permission to enter the park, and the like.

[0065] The machine learning model 132 can process the model inputs 138 to obtain the query image 140. In some implementations, the machine learning model 132 can include a machine learning language encoder model 208. The machine learning language encoder model 208 can be trained to perform various language tasks. In some implementations, the machine learning language encoder model 208 can be trained to process some or all of the model inputs 138 to obtain intermediate representations 210 of these inputs. The intermediate representation 210 can be any type or form of low-dimensional representation of the inputs to the machine learning language encoder model 208.

[0066] In some implementations, the machine learning language encoder model 208 may include a decoder model for performing a generative language task. In some implementations, the machine learning language encoder model 208 may be utilized to extract the visual descriptor 128 from the text content item 110. For example, the machine learning language encoder model 208 may process the text content item 110 and the prompt 202. The prompt 202 may include a request to extract the visual descriptor. For example, the prompt may include a request such as "describe the place in a single paragraph that can aid an artist to paint a picture of the place. Your description should use the visual elements mentioned in the reviews about the place given below. Be precise and succinct. (Describe this place in a single paragraph, so that it can help artists draw pictures of this place. Your description should use the visual elements mentioned in the reviews about this place given below. Be precise and succinct.)".

[0067] In some implementations, the machine learning model 132 may include a machine learning supplemental encoding model 209. The machine learning supplemental encoding model 210 may be any type or form of encoder model trained to process inputs and generate supplemental encodings 212 of those inputs. For example, assume that the model input 138 includes a recording of a user providing a verbal review of a POI. The machine learning supplemental encoding model 209 may process the recording to obtain a supplemental encoding 212 representing the recording.

[0068] The machine learning model 132 may include a machine learning generative visual model 214. The machine learning generative visual model 214 may be trained to process input to generate an image based on the input. More specifically, the input to the machine learning generative visual model 214 may describe a particular image, or may be an intermediate representation of a description of a particular image. The machine learning generative visual model 214 may process the input to obtain a query image 140, and the query image 140 may include the visual descriptor 128 processed by the machine learning generative visual model 214.

[0069] Following the depicted example, the machine learning language encoder model 208 can process the visual descriptors 128 for the POI (e.g., a ski hill) to obtain an intermediate representation 210 of the visual descriptors 128. The machine learning generative visual model 214 can process the intermediate representation 210 to obtain a query image 140. The query image 140 can depict the POI and some or all of the visual descriptors 128. Here, since the visual descriptors 128 include descriptions of skiing, snowboarding, and college students, the query image can depict a young person snowboarding on a snowy hill. The query image 140 can be provided to the visual similarity identifier 142.

[0070] Figure 3 is a block diagram 300 of a computing system that identifies representative images from a corpus of user-generated content based on visual similarity between the image and a query image, according to some implementations of the present disclosure. Figure 3 Will combine Figure 1 and Figure 2 Specifically, the computing system 102 may include a representation image recognizer 118, which may include a visual similarity recognizer 142. The visual similarity recognizer 142 may obtain a query image 140, such as Figure 2 described.

[0071] The visual similarity identifier 142 may also include a visual UGC corpus 144. The visual UGC corpus 144 may include candidate images 302A-302N (collectively, candidate images 302). The candidate images may be candidates for selection as representation images for the POI, and may include some or all of the UGC images 114 included in the visual UGC repository 112. Specifically, in some implementations, the visual similarity identifier 142 may filter some of the UGC images 114 so that they are not included in the visual UGC corpus as one of the candidate images 302. For example, the visual similarity identifier 142 may filter the UGC images 114 from the visual UGC corpus based on a quality metric associated with the image. For example, if the quality metric associated with the UGC image 114A is higher than a threshold quality metric, the UGC image 114A may be selected to be included in the visual UGC corpus 144 as the candidate image 302A. Conversely, if the quality metric associated with UGC image 114B is below the threshold quality metric, visual similarity identifier 142 may refrain from including UGC image 114B in visual UGC corpus 144. In this manner, visual similarity identifier 142 may reduce the computational cost of evaluating visual similarity by reducing the total number of images to evaluate.

[0072] The visual similarity identifier 142 may include an image similarity determiner 304. The image similarity determiner 304 may determine the visual similarity between the query image 140 and each of the candidate images 302. In some implementations, the image similarity determiner 304 may include a machine learning computer vision model 306, or if such a model is disposed by the machine learning module 130, the image similarity determiner may access the model. The machine learning computer vision model 306 may identify the degree of visual similarity between the images. In some implementations, the machine learning computer vision model 306 may be a model trained to perform semantic analysis of visual similarities between images. More specifically, the machine learning computer vision model 306 may be trained to determine the similarity between images based on a semantic understanding of the images. For example, candidate image 302A depicts a mountain biker, candidate image 302B depicts an icon representation of a person skiing, and candidate image 302N depicts a person snowboarding. Here, from a surface level view, query image 140 and candidate image 302A are more “visually similar” than query image 140 and candidate image 302B. However, the concept exemplified by candidate image 302B is more visually similar to query image 140 than the concept exemplified by candidate image 302A (e.g., biking vs. skiing). In this way, machine learning computer vision model 306 can determine visual similarity on a pixel-by-pixel basis and / or a concept basis.

[0073] In some implementations, the machine learning computer vision model 306 can generate embeddings 308A-308N (collectively, embeddings 308) for the query image 140 and the candidate images 302. Alternatively, in some implementations, the computing system 102 can obtain or otherwise access the image embeddings 308 and the candidate images 302, or can obtain the embeddings 308 and information associating the embeddings 308 with the candidate images 302.

[0074] As previously described, in some implementations, the machine learning generative vision model 214 can be utilized to generate an intermediate representation, such as an embedding, from which the query image 140 can be derived. In this way, the implementations described herein can retain the benefits of the query image 140 while significantly reducing the computational costs associated with generative image tasks.

[0075] For example, image similarity determiner 304 may obtain an embedding of query image 140. Image similarity determiner 304 may map the embedding of query image 140 to image embedding space 310. Image similarity determiner 304 may determine a distance between the embedding of query image 140 and image embedding 308. The distance between embeddings in image embedding space 310 may indicate a visual similarity between corresponding images, where embeddings that are located closer together are more visually similar than embeddings that are located farther apart. Thus, image similarity determiner 304 may identify candidate images 302 that are visually similar to representative image 146 by determining a distance between the embedding of query image 140 and the embedding of candidate image 302 within image embedding space 310.

[0076] For example, assume that embedding 308A represents query image 140, embedding 308B represents candidate image 302B, and embedding 308N represents candidate image 302N. If the distance between embeddings 308A and 308N within image embedding space 310 is less than the distance between embeddings 308A and 308B, image similarity determiner 304 may determine that candidate image 302N is more visually similar to query image 140 than candidate image 302B.

[0077] In some implementations, the visual similarity identifier 142 can select the representative image 146 based on visual similarity. Following the depicted example, the visual similarity identifier 142 can select the candidate image 302N as the representative image 146 based on a close visual similarity between the query image 140 and the candidate image 302N.

[0078] Additionally or alternatively, in some implementations, the visual similarity identifier 142 can generate a visual similarity signal 312. The visual similarity signal 312 can indicate a degree of visual similarity between the query image 140 and each of the candidate images 302. The visual similarity signal 312 can be one of a plurality of signals analyzed to select the representative image 146. For example, the representative image identifier 118 can include an image selector 314. The image selector 314 can select the candidate image 302N as the representative image 146 based on the visual similarity signal 312 and the additional signal 316.

[0079] Additional signals 316 may include any type or form of signal based on the content of candidate images 302, evaluation of candidate images 302, user interaction with candidate images 302, and the like. For example, additional signals 316 may include a feedback signal indicating user feedback received for some (or all) of candidate images 302. As another example, additional signals 316 may include a content signal indicating the extent of the likelihood that the content of the candidate image is inappropriate or sensitive. As another example, additional signals 316 may include a recency signal based on the length of time that has elapsed since a particular candidate image was captured. As another example, additional signals 316 may include a relevance signal indicating the extent of the relevance between the candidate image and the POI. As another example, additional signals 316 may include a privacy signal indicating the extent of the likelihood that the content of the candidate image presents a privacy risk (e.g., an image depicting the faces of multiple people may present a greater privacy risk than an image depicting a snowboarder whose entire face is covered). Image selector 314 may evaluate visual similarity signal 312 and additional signal 316 to identify candidate image 302N as representative image 146 .

[0080] Figure 4 is a block diagram 400 of a computing system that modifies a POI profile to replace an existing representation image with a newly selected representation image, according to some implementations of the present disclosure. Figure 4 Will combine Figures 1 to 3 Specifically, the computing system 102 may include Figure 1 The POI profile manager 106 may include a text UGC repository 108 , a visual UGC repository 112 , and a profile information repository 116 .

[0081] The POI profile manager 106 may include a profile modifier 402. The profile modifier 402 may generate a modification instruction 404 based on the representation image 146. More specifically, the profile modifier 402 may generate the modification instruction 404, which includes an instruction to modify an existing profile of the POI to replace a current representation image 410 with the representation image 146. For example, the profile information repository 116 may include profiles 406A-406N (collectively, profiles 406). Each of the profiles may include POI information 408 and a current representation image 410. The POI information 408 may include information to be presented to a user within the profile of the POI, such as business hours, a restaurant's menu, services offered, etc. The current representation image 410 may be a representation image currently displayed to a user within the profile of the POI. The modification instruction 404 may be provided to the profile information repository 116, and may instruct the profile information repository 116 to replace the current representation image 410 with the representation image 146. In this manner, computing system 102 may cause a newly identified representation image to replace an existing representation image.

[0082] Figure 5 An example profile for a particular POI within a map application associated with a map service is shown in accordance with some implementations of the present disclosure. Figure 5 Will combine Figures 1 to 4 Specifically, a user computing device 502 (e.g., a smart phone, laptop, wearable computing device, tablet computer, etc.) can execute an application 504. The application 504 can be associated with an application-based service (such as a navigation service or a map service). The application 504 can maintain a profile of various POIs that can be discovered via the application 504.

[0083] According to the depicted example, the user may select a ski area POI called "polar peaks". In response, the application-based service may send a profile 406B of the ski area to the user computing device 502. The profile 406B may be displayed within the application 504 at the user computing device 502. As shown, the representation image 146 may be placed in a prominent position 506 within the profile 406B. In this way, the representation image 146 may serve as a "first impression" to the user, which also provides a visual overview of the POI. In some implementations, the profile 406B may include current and / or previous representation images, such as the current representation image 410, and the previous representation image may be placed in a less prominent position (such as position 508) than the position 506.

[0084] Figure 6 A flowchart 600 is depicted for performing an example method of representing image selection according to an example embodiment of the present disclosure. Figure 6The operations are depicted as being performed in a particular order for purposes of illustration and discussion, but the methods of the present disclosure are not limited to the particular order or arrangement described. Various operations of method 600 may be omitted, rearranged in various ways, combined, and / or adjusted without departing from the scope of the present disclosure.

[0085] At operation 602, the computing system may extract one or more visual descriptors for a particular POI from a UGC corpus. The UGC corpus may include multiple text content items describing the particular POI. In some implementations, before extracting the visual descriptors, the computing system may detect the occurrence of a representation image evaluation event. In some implementations, the computing system may detect the passage of a predetermined time period after detecting a previous representation image evaluation event. In some implementations, the computing system may receive information indicating a request to evaluate a representation image. In some implementations, the computing system may determine that the seasonal conditions depicted in the representation image currently used to represent the particular POI are different from the current seasonal conditions.

[0086] Additionally or alternatively, in some implementations, before extracting the visual descriptors, the computing system may select a subset of text content items from the plurality of text content items based at least in part on user feedback information associated with the subset of text content items. The computing system may extract one or more visual descriptors for a specific POI from the subset of text content items. In some implementations, each text content item in the subset of text content items may include a comment on the specific POI written by a user.

[0087] In some implementations, the computing system may obtain user feedback information indicating a degree of representation accuracy of a representation image currently used to represent a specific POI. The computing system may determine that the degree of representation accuracy of the representation image currently used to represent the specific POI is less than a threshold degree of representation accuracy.

[0088] At operation 604, based on the one or more visual descriptors, the computing system may use a machine learning generative visual model to generate a query image depicting a particular POI. In some implementations, the computing system may use the model by processing the one or more visual descriptors with a language encoder model to obtain an intermediate representation. The computing system may use the machine learning generative visual model to process the intermediate representation of the one or more visual descriptors to generate the query image. In some implementations, the computing system may use the language encoder model to process the one or more visual descriptors and a set of context information to obtain an intermediate representation.

[0089] In some implementations, the context information set may include prompt information indicating instructions to generate a depiction of a particular POI including one or more visual descriptors. Additionally or alternatively, in some implementations, the context information set may include the context information set may include temporal context information. The temporal context information may indicate current seasonal conditions, current or predicted weather conditions, current or predicted traffic conditions, current or planned events associated with the particular POI, recent or planned modifications to the particular POI, recent or planned modifications to another POI associated with the particular POI, and the like.

[0090] At operation 606 , the computing system may determine a visual similarity between the query image and each candidate image of a plurality of candidate images of the visual UGC corpus that depict a particular POI.

[0091] At operation 608 , the computing system may select a representative image from among the plurality of images to represent the particular POI based on a visual similarity between the representative image and the query image.

[0092] Fig. 7A A block diagram of an example computing system 700 that performs selection of a representative image corresponding to a POI profile is depicted according to an example embodiment of the present disclosure. System 700 includes a user computing device 702, a server computing system 730, and a training computing system 750 communicatively coupled via a network 780.

[0093] The user computing device 702 can be any type of computing device, such as, for example, a personal computing device (e.g., a laptop or desktop computer), a mobile computing device (e.g., a smartphone or tablet), a game console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.

[0094] The user computing device 702 includes one or more processors 712 and a memory 714. The one or more processors 712 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.), and may be one processor or a plurality of processors operatively connected. The memory 714 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc., and combinations thereof. The memory 714 may store data 716 and instructions 718 that are executed by the processor 712 to cause the user computing device 702 to perform operations.

[0095] In some implementations, the user computing device 702 may store or include one or more models 720. For example, the model 720 may be or may otherwise include various machine learning models, such as a neural network (e.g., a deep neural network) or other types of machine learning models, including nonlinear models and / or linear models. The neural network may include a feedforward neural network, a recurrent neural network (e.g., a long short-term memory recurrent neural network), a convolutional neural network, or other forms of neural networks. Some example machine learning models may utilize attention mechanisms, such as self-attention. For example, some example machine learning models may include a multi-head self-attention model (e.g., a transformer model).

[0096] In some implementations, one or more models 720 may be received from a server computing system 730 via a network 780, stored in a user computing device memory 714, and then used or otherwise implemented by one or more processors 712. In some implementations, a user computing device 702 may implement multiple parallel instances of a single model 720 (e.g., to perform parallel generative tasks across multiple instances of a model).

[0097] Additionally or alternatively, one or more models 740 may be included in or otherwise stored and implemented by a server computing system 730 that communicates with the user computing device 702 according to a client-server relationship. For example, the model 740 may be implemented by the server computing system 730 as part of a web service (e.g., a map service, a visual search service, etc.). Thus, one or more models 720 may be stored and implemented at the user computing device 702, and / or one or more models 740 may be stored and implemented at the server computing system 730.

[0098] The user computing device 702 may also include one or more user input components 722 that receive user input. For example, the user input component 722 may be a touch-sensitive component (e.g., a touch-sensitive display screen or a touchpad) that is sensitive to the touch of a user input object (e.g., a finger or a stylus). The touch-sensitive component may be used to implement a virtual keyboard. Other example user input components include a microphone, a traditional keyboard, or other means by which a user can provide user input.

[0099] The server computing system 730 includes one or more processors 732 and a memory 734. The one or more processors 732 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.), and may be one processor or a plurality of processors operatively connected. The memory 734 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 734 may store data 736 and instructions 738, which are executed by the processor 732 to cause the server computing system 730 to perform operations.

[0100] In some implementations, the server computing system 730 includes or is otherwise implemented by one or more server computing devices. In instances where the server computing system 730 includes multiple server computing devices, such server computing devices may operate according to a sequential computing architecture, a parallel computing architecture, or some combination thereof.

[0101] As described above, the server computing system 730 may store or otherwise include one or more models 740. For example, the model 740 may be or may otherwise include various machine learning models. Example machine learning models include neural networks or other multi-layer nonlinear models. Example neural networks include feedforward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks. Some example machine learning models may utilize attention mechanisms, such as self-attention. For example, some example machine learning models may include a multi-head self-attention model (e.g., a transformer model). In particular, the model 740 may include information about Figure 1 and Figure 2 Machine learning models discussed132.

[0102] User computing device 702 and / or server computing system 730 may train models 720 and / or 740 via interaction with training computing system 750 communicatively coupled via network 780. Training computing system 750 may be separate from server computing system 730 or may be part of server computing system 730.

[0103] The training computing system 750 includes one or more processors 752 and a memory 754. The one or more processors 752 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.), and can be one processor or multiple processors operatively connected. The memory 754 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc. and combinations thereof. The memory 754 can store data 756 and instructions 758, which are executed by the processor 752 to cause the training computing system 750 to perform operations. In some implementations, the training computing system 750 includes one or more server computing devices or is otherwise implemented by one or more server computing devices.

[0104] The training computing system 750 may include a model trainer 760 that trains the machine learning models 720 and / or 740 stored at the user computing device 702 and / or the server computing system 730 using various training or learning techniques such as, for example, error back propagation. For example, a loss function may be back propagated through the model to update one or more parameters of the model (e.g., based on the gradient of the loss function). Various loss functions may be used, such as mean squared error, likelihood loss, cross entropy loss, hinge loss, and / or various other loss functions. Gradient descent techniques may be used to iteratively update parameters over a number of training iterations.

[0105] In some implementations, performing backpropagation of errors may include performing truncated backpropagation through time. Model trainer 760 may perform a number of generalization techniques (e.g., weight decay, dropout, etc.) to improve the generalization capabilities of the model being trained. In particular, model trainer 760 may train models 720 and / or 740 based on a set of training data 762. Alternatively, in some implementations, machine learning model 132 may be pre-trained, and thus computing system 730 may avoid performing additional training iterations.

[0106] In some implementations, if the user has provided consent, the training examples may be provided by the user computing device 702. Thus, in such implementations, the model 720 provided to the user computing device 702 may be trained by the training computing system 750 based on user-specific data received from the user computing device 702. In some instances, this process may be referred to as personalizing the model.

[0107] The model trainer 760 includes computer logic utilized to provide the desired functionality. The model trainer 760 may be implemented in hardware, firmware, and / or software that controls a general purpose processor. For example, in some implementations, the model trainer 760 includes a program file stored on a storage device, loaded into a memory, and executed by one or more processors. In other implementations, the model trainer 760 includes one or more computer executable instruction sets stored in a tangible computer readable storage medium such as a RAM, a hard disk, or an optical or magnetic medium.

[0108] The network 780 can be any type of communication network, such as a local area network (e.g., an intranet), a wide area network (e.g., the Internet), or some combination thereof, and can include any number of wired or wireless links. In general, communications over the network 780 can be conducted via any type of wired and / or wireless connection using a wide variety of communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and / or protection schemes (e.g., VPN, secure HTTP, SSL).

[0109] The machine learning models described in this specification can be used for a variety of tasks, applications, and / or use cases.

[0110] In some implementations, the input of the machine learning model of the present disclosure may be image data. The machine learning model may process the image data to generate an output. As an example, the machine learning model may process the image data to generate an image recognition output (e.g., recognition of image data, potential embedding of image data, encoded representation of image data, hash of image data, etc.). As another example, the machine learning model may process the image data to generate an image segmentation output. As another example, the machine learning model may process the image data to generate an image classification output. As another example, the machine learning model may process the image data to generate an image data modification output (e.g., a change in image data, etc.). As another example, the machine learning model may process the image data to generate an encoded image data output (e.g., an encoded representation and / or compressed representation of image data, etc.). As another example, the machine learning model may process the image data to generate an amplified image data output. As another example, the machine learning model may process the image data to generate a prediction output.

[0111] In some implementations, the input of the machine learning model of the present disclosure may be text or natural language data. The machine learning model may process the text or natural language data to generate an output. As an example, the machine learning model may process the natural language data to generate a language encoding output. As another example, the machine learning model may process the text or natural language data to generate a potential text embedding output. As another example, the machine learning model may process the text or natural language data to generate a translation output. As another example, the machine learning model may process the text or natural language data to generate a classification output. As another example, the machine learning model may process the text or natural language data to generate a text segmentation output. As another example, the machine learning model may process the text or natural language data to generate a semantic intent output. As another example, the machine learning model may process the text or natural language data to generate an amplified text or natural language output (e.g., text or natural language data with higher quality than the input text or natural language, etc.). As another example, the machine learning model may process the text or natural language data to generate a prediction output.

[0112] In some implementations, the input of the machine learning model of the present disclosure may be speech data. The machine learning model may process the speech data to generate an output. As an example, the machine learning model may process the speech data to generate a speech recognition output. As another example, the machine learning model may process the speech data to generate a speech translation output. As another example, the machine learning model may process the speech data to generate a potential embedding output. As another example, the machine learning model may process the speech data to generate an encoded speech output (e.g., an encoded representation and / or compressed representation of the speech data, etc.). As another example, the machine learning model may process the speech data to generate an amplified speech output (e.g., speech data of higher quality than the input speech data, etc.). As another example, the machine learning model may process the speech data to generate a text representation output (e.g., a text representation of the input speech data, etc.). As another example, the machine learning model may process the speech data to generate a predicted output.

[0113] In some implementations, the input of the machine learning model of the present disclosure may be latent coded data (e.g., a latent space representation of an input, etc.). The machine learning model may process the latent coded data to generate an output. As an example, the machine learning model may process the latent coded data to generate a recognition output. As another example, the machine learning model may process the latent coded data to generate a reconstruction output. As another example, the machine learning model may process the latent coded data to generate a search output. As another example, the machine learning model may process the latent coded data to generate a re-clustering output. As another example, the machine learning model may process the latent coded data to generate a prediction output.

[0114] In some cases, the input includes visual data, and the task is a computer vision task. In some cases, the input includes pixel data for one or more images, and the task is an image processing task. For example, the image processing task may be image classification, where the output is a set of scores, each score corresponding to a different object class and representing the likelihood that one or more images depict an object belonging to the object class. The image processing task may be object detection, where the image processing output identifies one or more regions in one or more images, and for each region, identifies the likelihood that the region depicts an object of interest. As another example, the image processing task may be image segmentation, where the image processing output defines the corresponding likelihood of each category in a predetermined set of categories for each pixel in one or more images. For example, the set of categories may be foreground and background. As another example, the set of categories may be object classes. As another example, the image processing task may be depth estimation, where the image processing output defines a corresponding depth value for each pixel in one or more images. As another example, the image processing task may be motion estimation, where the network input includes multiple images, and the image processing output defines the motion of the scene depicted at the pixel between the images in the network input for each pixel in one of the input images.

[0115] In some cases, the input includes audio data representing a spoken utterance, and the task is a speech recognition task. The output may include a text output mapped to the spoken utterance. In some cases, the task includes encrypting or decrypting the input data. In some cases, the task includes a microprocessor performance task, such as branch prediction or memory address translation.

[0116] Fig. 7A An example computing system that can be used to implement the present disclosure is shown. Other computing systems may also be used. For example, in some implementations, the user computing device 702 may include a model trainer 760 and a training data set 762. In such implementations, the model 720 may be trained and used locally at the user computing device 702. In some of such implementations, the user computing device 702 may implement the model trainer 760 to personalize the model 720 based on user-specific data.

[0117] Figure 7B A block diagram of an example computing device 770 is depicted for training a model for representing image selections according to an example embodiment of the present disclosure. The computing device 770 may be a user computing device or a server computing device.

[0118] Computing device 770 includes several applications (e.g., applications 1 to N). Each application contains its own machine learning library and machine learning model. For example, each application may include a machine learning model. Example applications include text messaging applications, email applications, dictation applications, virtual keyboard applications, browser applications, etc.

[0119] like Figure 7B As shown, each application can communicate with a number of other components of the computing device (such as, for example, one or more sensors, a context manager, a device state component, and / or additional components). In some implementations, each application can communicate with each device component using an API (e.g., a public API). In some implementations, the API used by each application is specific to the application.

[0120] Figure 7C A block diagram of an example computing device 785 utilizing a machine learning model for representation image selection is depicted according to an example embodiment of the present disclosure. The computing device 785 may be a user computing device or a server computing device.

[0121] The computing device 785 includes a number of applications (e.g., applications 1 to N). Each application communicates with a central intelligence layer. Example applications include text messaging applications, email applications, dictation applications, virtual keyboard applications, browser applications, etc. In some implementations, each application can communicate with the central intelligence layer (and the models stored therein) using an API (e.g., a public API across all applications).

[0122] The central intelligence layer includes several machine learning models. For example, Figure 7C As shown, a corresponding machine learning model can be provided for each application, and the corresponding machine learning model can be managed by the central intelligence layer. In other implementations, two or more applications can share a single machine learning model. For example, in some implementations, the central intelligence layer can provide a single model for all applications. In some implementations, the central intelligence layer is included in the operating system of the computing device 785 or is otherwise implemented by the operating system.

[0123] The central intelligence layer can communicate with the central device data layer. The central device data layer can be a centralized data repository for computing devices 785. Figure 7C As shown, the central device data layer can communicate with several other components of the computing device (such as, for example, one or more sensors, a context manager, a device state component, and / or additional components). In some implementations, the central device data layer can use an API (e.g., a private API) to communicate with each device component.

[0124] The technology discussed herein relates to servers, databases, software applications, and other computer-based systems, as well as the actions taken and the information sent to and from such systems. The inherent flexibility of computer-based systems allows for multiple possible configurations, combinations, and divisions of tasks and functions between and among components. For example, the processes discussed herein can be implemented using a single device or component or multiple devices or components working in combination. Databases and applications can be implemented on a single system, or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0125] Although the subject matter has been described in detail with respect to various specific example embodiments of the subject matter, each example is provided by way of explanation, not limitation of the present disclosure. Those skilled in the art can easily produce changes to such embodiments, changes to such embodiments, and equivalents to such embodiments after understanding the foregoing. Therefore, the present disclosure does not exclude such modifications, changes, and / or additions to the subject matter that will be readily understood by those of ordinary skill in the art. For example, a feature shown or described as part of one embodiment may be used together with another embodiment to produce yet another embodiment. Therefore, the present disclosure is intended to cover these changes, changes, and equivalents.

Claims

1. A computer-implemented method comprising: Extracting, by a computing system including one or more processor devices, one or more visual descriptors for a specific point of interest (POI) from a user-generated content (UGC) corpus, wherein the UGC corpus includes a plurality of text content items describing the specific POI; generating, by the computing system, a query image depicting the specific POI using a machine learning generative vision model based on the one or more visual descriptors; determining, by the computing system, a visual similarity between the query image and each of a plurality of candidate images of a visual UGC corpus depicting the specific POI; as well as The representative image is selected by the computing system from the plurality of candidate images to represent the particular POI based on the visual similarity between the representative image and the query image.

2. The computer-implemented method of claim 1 , wherein generating the query image using the machine learning generative vision model comprises: processing, by the computing system, the one or more visual descriptors using a language encoder model to obtain an intermediate representation; as well as The intermediate representation of the one or more visual descriptors is processed by the computing system using the machine learning generative vision model to generate the query image.

3. The computer-implemented method of claim 2, wherein processing the one or more visual descriptors using the language encoder model to obtain the intermediate representation comprises: The one or more visual descriptors and the set of context information are processed by the computing system using the language encoder model to obtain the intermediate representation. 4 . The computer-implemented method of claim 3 , wherein the set of context information includes prompt information indicating instructions to generate a depiction of the particular POI including the one or more visual descriptors.

5. The computer-implemented method of claim 4, wherein the set of context information includes temporal context information indicating at least one of: Current seasonal conditions; current or forecasted weather conditions; current or forecasted traffic conditions; current or planned events associated with the particular POI; any recent or planned modifications to the particular POI; or Recent or planned modifications to another POI associated with the particular POI.

6. The computer-implemented method of claim 5, wherein processing the one or more visual descriptors and the set of context information comprises: processing, by the computing system, the one or more visual descriptors and the temporal context information using the language encoder model to obtain the intermediate representation, wherein the temporal context information describes a current season; and Wherein selecting the representative image from the plurality of candidate images comprises: The representative image is selected, by the computing system, from the plurality of candidate images to represent the specific POI based on the visual similarity between the representative image and the query image, wherein the representative image includes a depiction of the specific POI under current seasonal conditions, and wherein an existing representative image currently used to represent the specific POI depicts the specific POI under seasonal conditions other than the current seasonal conditions.

7. The computer-implemented method of claim 6, wherein prior to extracting the one or more visual descriptors, the method comprises: An occurrence representing an image evaluation event is detected by the computing system.

8. The computer-implemented method of claim 7, wherein detecting the occurrence of the representative image evaluation event comprises: The lapse of a predetermined period of time after the detection of the preceding representative image evaluation event is detected by the computing system.

9. The computer-implemented method of claim 7, wherein detecting the occurrence of the representative image evaluation event comprises: Information indicative of a request to evaluate the representative image is received by the computing system.

10. The computer-implemented method of claim 7, wherein detecting the occurrence of the representative image evaluation event comprises: It is determined by the computing system that seasonal conditions depicted in the representative image currently used to represent the particular POI are different from current seasonal conditions.

11. The computer-implemented method of claim 7, wherein detecting the occurrence of the representative image evaluation event comprises: obtaining, by the computing system, user feedback information indicating a degree of representation accuracy of the representation image currently used to represent the specific POI; as well as The computing system determines that the degree of representation accuracy of the representation image currently used to represent the specific POI is less than a threshold degree of representation accuracy.

12. The computer-implemented method of claim 1, wherein the method further comprises: The computing system causes the representation image to be used to replace a representation image currently used by a service to represent the specific POI.

13. The computer-implemented method of claim 12, wherein the service comprises a mapping service and / or a visual search service.

14. The computer-implemented method of claim 1 , wherein extracting the one or more visual descriptors for the particular POI comprises: selecting, by the computing system, the subset of textual content items from the plurality of textual content items based at least in part on user feedback information associated with the subset of textual content items; as well as The one or more visual descriptors for the specific POI are extracted by the computing system from the subset of text content items.

15. The computer-implemented method of claim 14, wherein each text content item in the subset of text content items comprises a review of the particular POI written by a user.

16. A computing system comprising: one or more processor devices; one or more tangible, non-transitory computer-readable media storing computer-readable instructions that, when executed by the one or more processor devices, cause the one or more processor devices to perform operations comprising: Extracting one or more visual descriptors for a specific point of interest (POI) from a user-generated content (UGC) corpus, wherein the UGC corpus includes a plurality of text content items describing the specific POI; generating a query image depicting the specific POI using a machine learning generative vision model based on the one or more visual descriptors; determining a visual similarity between the query image and each of a plurality of candidate images of a visual UGC corpus depicting the particular POI; and The representative image among the plurality of candidate images is selected to represent the specific POI based on the visual similarity between the representative image and the query image.

17. The computing system of claim 16, wherein using the machine learning generative vision model to generate the query image comprises: processing the one or more visual descriptors using a language encoder model to obtain an intermediate representation; as well as The intermediate representation of the one or more visual descriptors is processed using the machine learning generative vision model to generate the query image.

18. The computing system of claim 17, wherein processing the one or more visual descriptors using the language encoder model to obtain the intermediate representation comprises: The one or more visual descriptors and the set of context information are processed using the language encoder model to obtain the intermediate representation.

19. The computing system of claim 18, wherein the set of context information includes hint information indicating instructions to generate a depiction of the particular POI including the one or more visual descriptors.

20. One or more tangible non-transitory computer-readable media storing computer-readable instructions that, when executed by one or more processor devices, cause the one or more processor devices to perform operations comprising: Extracting one or more visual descriptors for a specific point of interest (POI) from a user-generated content (UGC) corpus, wherein the UGC corpus includes a plurality of text content items describing the specific POI; generating an embedding using a machine learning generative vision model based on the one or more visual descriptors, from which a query image depicting the particular POI can be generated; determining a similarity between the embedding and each of a plurality of embeddings respectively representing a plurality of candidate images of a visual UGC corpus depicting the particular POI; as well as A representative image from the plurality of candidate images is selected to represent the particular POI based on the similarity between the embedding and each of the plurality of embeddings.

Images