Generating prompts for user link annotations

Abstract

Systems and methods for generating prompts for user data entry may include obtaining contextual data. Contextual data may be processed to determine whether an input entry interface is to be provided. In response to determining that the input entry interface is to be provided, the contextual data or other data associated with the content display instance may be processed with a generative model to generate a prompt, which may be provided to the user. User input data may then be obtained and stored for provision to other users.

Description

[0001] Related Applications

[0002] This application is based on and claims priority to U.S. non-provisional application No. 18 / 392,648, filed on December 21, 2023, which claims the benefit of U.S. provisional application serial number 63 / 596,484, filed on November 6, 2023. Applicants claim priority to and the benefit of each of such applications, and all of such applications are incorporated herein by reference in their entirety. Technical Field

[0003] The present disclosure generally relates to generating prompts for obtaining link annotations. More specifically, the present disclosure relates to determining when and how to prompt a user to provide annotations about links associated with web resources, which annotations can then be provided to other users. Background Art

[0004] Understanding search results from search result pages can be difficult because titles and text snippets may provide limited information that may not be relevant to the user's interests, which can result in time-consuming web resource consultations that may not produce the desired information. Obtaining additional information about web resources can be difficult, which may include additional searches that may or may not identify relevant information.

[0005] Additionally, obtaining user insights may be difficult. Specifically, it may be difficult for users to determine which words to use. Additionally, these words may not be targeted to other users' interests and / or may not be rich enough to generate the desired results. Summary of the invention

[0006] Aspects and advantages of embodiments of the present disclosure will be set forth in part in the following description, or may be learned from the description, or may be learned through practice of the embodiments.

[0007] An example aspect of the present disclosure relates to a computing system for comment prompt generation and input retrieval, which may include: one or more processors; and one or more non-transitory computer-readable media, which together store instructions that, when executed by the one or more processors, cause the computing system to perform operations. The operations may include: obtaining content data. The content data may be associated with a web resource. The operations may include: processing the content data with a generative model to generate a predictive prompt. The prompt may include a predictive text string associated with commenting on a web resource. The operations may include: providing the predictive prompt for display with an input prompt interface. The input prompt interface may be configured to receive input. The operations may include: obtaining comment input data from a user computing system via the input prompt interface. In some implementations, the comment input data may include user-generated comments on a web resource. The operations may include: storing data associated with the comment input data together with data related to the web resource. The data associated with the comment input data may be stored in a searchable database to be provided for display in response to the web resource being provided as a search result.

[0008] In some implementations, operations may include obtaining user data. The user data may be associated with a specific user. The user computing system may be associated with a specific user. Processing content data with a generative model to generate a predictive prompt may include processing content data and user data with a generative model. The user data may include user search history data. The generative model may generate predictive prompts based on a specific user's previous search for information associated with a theme of a web resource. In some implementations, the user data may include user browser history data. The generative model may generate predictive prompts based on a specific user's previous viewing of other web resources including information associated with a theme of a web resource. Operations may include generating a graphics card based on user data, content data, and comment input data. The graphics card may include a user profile identifier for a specific user and data associated with the comment input data. Operations may include storing the graphics card. The graphics card may include a graphic background generated based on the comment input data using an image generation model.

[0009] In some implementations, operations may include obtaining a search query, determining that a web resource is associated with the search query, and providing specific search results for display. The specific search results may include a link to the web resource, a title of the web resource, and data associated with the comment input data. Storing the data associated with the comment input data together with the data associated with the web resource may include generating a web resource annotation, and storing the web resource annotation together with a plurality of other web resource annotations associated with the web resource. In some implementations, operations may include providing the web resource annotation and a plurality of other web resource annotations in an annotation interface that provides the web resource annotation and a plurality of other web resource annotations in a plurality of graphics cards. The generation model may include an autoregressive language model. The generation model may be prompted to generate questions describing a request for information about the web resource.

[0010] Another example aspect of the present disclosure relates to a computer-implemented method for link annotation prompts. The method may include: obtaining context data by a computing system including one or more processors. The context data may be associated with a specific content display instance. The specific content display instance may include a specific user viewing a specific content item. The method may include: determining an input request action based on the context data by the computing system. The input request action may include providing an input entry interface to a user to obtain user input. The method may include: processing the context data with a generated language by the computing system to generate a prediction prompt. In some implementations, the prediction prompt may include a natural language request for information generated based on the context data. The method may include: providing a prediction prompt in an input entry interface by the computing system, and obtaining user-generated content by the computing system via the input entry interface. The method may include: generating a link annotation by the computing system based on the user-generated content. The link annotation may be generated to be provided for display in the search result interface in response to a specific content item being determined as a search result.

[0011] In some implementations, context data may be associated with the type of content being provided for display. Context data may be associated with a specific user associated with a specific content display instance. Context data may include search history data. Content being provided for display may be associated with a specific web resource. Context data may be associated with interaction data for links to specific web resources on multiple social networking platforms. An input request action may be determined based on interaction data. In some implementations, context data may include user data and content data. An input request action may be determined based on whether a topic associated with the content being provided for display is one of multiple topics that a specific user is aware of based on user data. Context data may include previous annotations generated by a specific user. Predictive prompts may include structures based on previous structures of previous annotations.

[0012] Another example aspect of the present disclosure relates to one or more non-transitory computer-readable media that collectively store instructions that, when executed by one or more computing devices, cause one or more computing devices to perform operations. The operations may include obtaining a first search query at a first time and determining that a web resource is responsive to the first search query. The operations may include obtaining content data. The content data may be associated with a web resource. The operations may include processing the content data with a generative model to generate a predictive prompt. The prompt may include a predictive text string associated with commenting on a web resource. The operations may include providing a predictive prompt for display within an input prompt interface. The input prompt interface may include an input entry box. The operations may include obtaining comment input data from a user computing system via the input prompt interface. The comment input data may include user-generated content. The operations may include storing user-generated content. The operations may include obtaining a second search query at a second time. The second time may be different from the first time. The operations may include determining that a web resource is responsive to a second search query and providing user-generated content in a search result interface having data describing the web resource.

[0013] Another example aspect of the present disclosure relates to a computing system for graphics card generation. The system may include: one or more processors; and one or more non-transitory computer-readable media, which together store instructions that, when executed by the one or more processors, cause the computing system to perform operations. The operations may include: obtaining card data. The card data may describe the content in the graphics card. The content may be associated with one or more topics. The operations may include: processing the card data to determine one or more entity tags associated with the content. The one or more entity tags may be associated with one or more topics. The operations may include: accessing a media content item database to obtain one or more media content items. The one or more media content items may be obtained based on determining that the one or more media content items are associated with one or more entity tags associated with the content. The operations may include: providing one or more media content items for display. The one or more media content items may be provided for display in an interactive user interface. The one or more media content items may be selectable to be inserted into the graphics card.

[0014] In some implementations, the graphics card may be associated with a link annotation. The link annotation may include user-generated content tagged to a specific web resource. The operation may include obtaining an input selection associated with one or more media content items, generating an enhanced graphics card, and providing the enhanced graphics card for display. The enhanced graphics card may include at least a portion of the content of the graphics card and at least a portion of the one or more media content items. In some implementations, the operation may include obtaining an adjustment input. The adjustment input may be associated with a request to enhance the enhanced graphics card. The operation may include generating an updated graphics card based on the adjustment input. The updated graphics card may include an enhanced graphics card that has been adjusted by one or more adjustments. The operation may include providing an updated graphics card for display. The one or more adjustments may include at least one of a layout change of the enhanced graphics card, a crop change of one or more media content items, a size change of one or more content items, a color change, or a template change.

[0015] In some implementations, the media content item database may include a user-specific database. The user-specific database may be associated with a specific user. The specific user may have generated at least a portion of the content. In some implementations, the user-specific database may include an image gallery associated with the specific user. The image gallery may be stored on a server computing system associated with a specific content item storage platform. In some implementations, the user-specific database may include a local storage database of a user computing device. The media content item database may include a plurality of media content items. In some implementations, a plurality of content items may have been pre-processed to generate a plurality of corresponding metadata sets.

[0016] Another example aspect of the present disclosure relates to a computing system. The system may include: one or more processors; and one or more non-transitory computer-readable media, which together store instructions that, when executed by the one or more processors, cause the computing system to perform operations. The operations may include: providing an input drafting interface for display. The input drafting interface may include a graphical user interface that includes multiple attribute options and a text input box. The multiple attribute options may be associated with multiple candidate attributes for content item generation. The operations may include: obtaining a selection of a specific attribute option from the multiple attribute options via the input drafting interface. The specific attribute option may be associated with a specific candidate attribute. The operations may include: obtaining text input via the text input box of the input drafting interface. The text input may be associated with a suggestion diagram for content item generation. The operations may include: processing the specific attribute option and the text input with a generation model to generate a model-generated content item. The model-generated content item may include a specific candidate attribute. In some implementations, the model-generated content item may be associated with a suggestion diagram. The operations may include: providing the model-generated content item for display via the input drafting interface.

[0017] In some implementations, the operations may further include obtaining an input selection via an input drafting interface, and generating an enhanced graphics card based on the input selection. The enhanced graphics card may include a graphics card enhanced to include a content item generated by the model. The operations may include providing the enhanced graphics card for display. The plurality of candidate attributes may include a plurality of different styles. The plurality of different styles may be associated with at least one of a plurality of different artistic styles or a plurality of different writing styles.

[0018] In some implementations, the plurality of candidate attributes may include a plurality of different tones. The plurality of different tones may be associated with at least one of a plurality of different emotions or a plurality of different rhythm types. In some implementations, the generative model may be obtained from a generative model database based on a selection of a specific attribute option. The specific attribute soft prompt may be obtained based on a selection of the specific attribute option. The specific attribute soft prompt may include a learned parameter set. The learned parameter set may be processed with the generative model to generate a model-generated content item.

[0019] In some implementations, the first search query and the second search query may be different. The review input data may include multimodal data. The multimodal data may include text data and image data.

[0020] Other aspects of the disclosure relate to various systems, apparatuses, non-transitory computer-readable media, user interfaces, and electronic devices.

[0021] These and other features, aspects and advantages of various embodiments of the present disclosure will be better understood with reference to the following description and appended claims.The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate example embodiments of the present disclosure and, together with the description, serve to explain the relevant principles. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] A detailed discussion of embodiments for those of ordinary skill in the art is set forth in this specification with reference to the accompanying drawings, in which:

[0023] Figure 1 Depicted is a block diagram of an example link annotation generation system according to an example embodiment of the present disclosure.

[0024] Figure 2 Depicted is a block diagram of an example user prompting system according to an example embodiment of the present disclosure.

[0025] Figure 3 Depicted is a flow chart of an example method for performing link annotation prompting according to an example embodiment of the present disclosure.

[0026] Figure 4 Depicted are diagrams of example prompts according to example embodiments of the present disclosure.

[0027] Figure 5A Depicted is a diagram of an example annotation interface with topic prompts according to an example embodiment of the present disclosure.

[0028] Figure 5B Depicted is an illustration of an example annotation interface with similar article prompts according to an example embodiment of the present disclosure.

[0029] Figure 5C Depicted is a diagram of an example prediction hint according to an example embodiment of the present disclosure.

[0030] FIG. 6A to FIG. 6C Depicted is a diagram of an example link annotation entry point according to an example embodiment of the present disclosure.

[0031] Figure 7 Depicted is a flow chart of an example method for performing link annotation generation according to an example embodiment of the present disclosure.

[0032] Figure 8 Depicted is a flow chart of an example method for performing link annotation display according to an example embodiment of the present disclosure.

[0033] FIG. 9A to FIG. 9G Depicted is a diagram of an example graphics card interface according to an example embodiment of the present disclosure.

[0034] Fig.10Depicted is a diagram of an example card generation interface according to an example embodiment of the present disclosure.

[0035] Fig.11 Depicted is a diagram of an example content item generation interface according to an example embodiment of the present disclosure.

[0036] Fig.12 Depicted is a diagram of an example image suggestion interface according to an example embodiment of the present disclosure.

[0037] Fig.13 Depicted is a flow chart of an example method for performing image suggestions according to an example embodiment of the present disclosure.

[0038] Fig.14 Depicted is a flow chart of an example method for performing content item generation according to an example embodiment of the present disclosure.

[0039] Fig.15A Depicted is a block diagram of an example computing system that performs link annotation hinting according to an example embodiment of the present disclosure.

[0040] Fig. 15B Depicted is a block diagram of an example computing system that performs link annotation hinting according to an example embodiment of the present disclosure.

[0041] Reference numerals that are repeated across multiple drawings are intended to identify like features in various implementations. DETAILED DESCRIPTION

[0042] In general, the present disclosure relates to generating prompts for user data entry. Specifically, the systems and methods disclosed herein may utilize context determination (e.g., determining a context in which a user is likely to provide an annotation and / or determining a comment gap and / or content gap for a particular link) to determine to provide an input entry interface (e.g., a link annotation input entry interface), and may utilize a generative model (e.g., a large language model) to generate prompts based on user data (e.g., user search history and / or user browsing history) and / or content data (e.g., the subject of the content and / or the type of content). For example, a user may be prompted to provide annotations about a particular web resource (and / or other content item) in a search results page, during a web resource review, and / or at the next search instance. Prompts may be generated based on previous user annotations, previously viewed content, the subject of the content, and / or the type of content to provide the user with prompts requesting information in a format that results in the generation of insightful annotations.

[0043] Link annotations can provide additional information about a web resource without consulting the web resource, and link annotations can be provided by other users. Systems and methods can determine when to provide link annotation prompts to users based on contexts that are determined to be associated with the intake of valuable annotations. For example, a particular user can provide more reliable and / or more detailed information about a particular topic based on previously acquired knowledge and / or based on previously generated annotations. Additionally and / or alternatively, a particular content type can be determined to be associated with user comments and / or user confusion.

[0044] The prompts provided to the user can "inspire" the user to provide more detailed information, and / or can guide the user to leave notes about specific topics and / or features of the web resource. The generation model can process user data and / or content data to generate predictive prompts. Specifically, the generation model can utilize the user's search history, the user's browsing history, the user's previous notes, and / or other user data to generate suggested notes, questions to prompt responses, and / or note templates. Alternatively and / or additionally, the generation model can utilize semantic understanding of the web resource, subject classification, content type classification, other notes associated with the web resource, and / or other content data to generate suggested notes, questions to prompt responses, and / or note templates.

[0045] The input entry interface may provide a prediction prompt to the user. The input entry interface may then obtain input (e.g., comment input data) from the user to generate user-generated content describing the link annotation. In some implementations, a graphic card may be generated based on the link annotation. The graphic card may include the user-generated content of the link annotation, a user profile identifier (e.g., name and / or image), link information, and / or a graphic background. The link annotation and / or graphic card may be stored in association with a web resource. The stored link annotation and / or graphic card may then be obtained in response to one or more users searching for a web resource and / or one or more users interacting with the annotation interface.

[0046] It may be difficult to understand the search results from the search results page because the titles and text snippets may provide limited information that may not be relevant to the user's interests, which may result in time-consuming web resource consultations that may not produce the desired information. Additionally and / or alternatively, obtaining additional information about the web resource may be difficult, which may include additional searches that may or may not identify relevant information. Social media posts, blog posts, and / or reviews of the web resource and / or entities associated with the web resource may lack details, may be misleading, and / or may lack context and / or perspective.

[0047] Link annotations (e.g., link annotations obtained from users and / or generated by a generative model) can provide additional information about a web resource, which can inform other users of its relevance to their requests. Link annotations can be provided in a search results page and / or can be displayed in an annotation interface accessible from a search results page and / or from a web resource. Link annotations can be provided in a graphic card, a text panel inline with a text snippet, and / or other formats.

[0048] Determining when and how to generate prompts for annotations can be based on a determination of annotation differences (e.g., whether there are a large number of annotations related to the article on the blog platform and / or social media platform, but relatively few in the annotation interface of the search platform), a determination of user-specific interests (e.g., whether the resource is similar to other articles that the user has viewed in the past), resource trends (e.g., whether the resource and / or similar resources have been previously reviewed), a determination of resources that are worth annotating (e.g., does the annotation provide utility?), and / or other determinations. The prompts can be generated based on generative language model processing, which can include processing previous annotations (e.g., other annotations made by the user and / or other users), processing search queries, processing web resources, and / or processing other data.

[0049] Link annotation prompts may be utilized to initiate and / or cause the collection of information about web resources, which may then be provided to other users, who may identify user opinions, user summaries, and / or other user-identified details. The obtained annotations may then be provided in a search results interface and / or a discover feed. Link annotation prompts may be determined based on determinations of annotation differences, determinations of user-specific interests, resource trends, determinations of resources worth annotating, and / or other determinations.

[0050] Obtaining additional information from other users may be useful for users to determine the subject matter and quality of search results that may not be discernible from traditional search result displays; however, obtaining helpful and detailed information may be difficult. Determining when to generate prompts for annotations and generating context-aware prompts can be used to provide an interface for obtaining detailed information from related users.

[0051] It can be difficult to motivate users to contribute information (e.g., comments, reviews, insights, etc.) when they see search results. A prompt generation system can be utilized to target the right prompt to the right user at the right time / place based on insights. Specifically, prompts can help create posts with desired characteristics (e.g., desired level of detail, desired topic, and / or other characteristics). The prompt generation system disclosed herein can facilitate the generation (or creation) of annotations by specific users, and / or can facilitate the generation (or creation) of annotations for specific web resources (and / or content items) via the prompt generation system.

[0052] In some implementations, the systems and methods disclosed herein may be utilized to prompt users to generate self-reliant content. Self-reliant content may include user recipes, user tutorials, user graphics, life updates, link sharing, and / or other user-generated content. Self-reliant content may be generated in a free form, and / or based on model-generated prompts. In some implementations, one or more machine learning models may be utilized to generate content templates, and / or one or more machine learning models may be utilized to enhance user-provided content (e.g., restructure and / or redesign text, images, audio, interface elements, and / or video).

[0053] In some implementations, link annotations and / or interactions with link annotations can be used to adjust web resource rankings, web resource tags, web resource embeddings, and / or web resource indexing. For example, in some implementations, link annotations can be processed to determine the quality of a web resource. The quality determination can be determined based on processing link annotations using one or more machine learning models (e.g., sentiment analysis models, language models, classification models, etc.). Link annotations can be processed using one or more machine learning models to determine topics associated with web resources, determine biases of web resources, the usefulness of web resources, and / or the direction of web resources. Link annotations can be used to suggest additional content, can be embedded for embedding-based searches, and / or can be used for query suggestions.

[0054] Link annotations in the annotation interface may be ranked and / or displayed based on interactions, quality determined by a machine learning model, responsiveness to a query, level of detail, and / or other attributes. In some implementations, link annotations generated by a user may be provided to all other users, only to users within the user's social network, and / or only to users determined to be associated with the user based on interests, location, and / or activities.

[0055] Link annotations can be used for a number of different content items and may not be limited to web resources. For example, the systems and methods disclosed herein can be used to generate prompts and / or interfaces for obtaining, eliciting, and / or generating link annotations for local files (e.g., documents, images, videos, etc. on a device), intranet files, and / or other content item sources, which may include folders on external drives, documents on the cloud, etc.

[0056] In some implementations, the input interface may include an open input interface that provides one or more options for providing user input. Alternatively and / or additionally, the input interface may include multiple features and / or options for generating user-generated content, which may be used for link annotations and / or stand-alone content. The input interface may include an independent content item user interface that may enable a user to add images, links, and / or content of different template types and may be interactive. The interactive user interface may include image suggestions, template suggestions, text suggestions, layout suggestions, link suggestions, widget suggestions, template suggestions, and / or other options (e.g., other types of suggestions).

[0057] Image suggestions may include processing user input text, data associated with a web resource, a generated prompt, a stock image library, and / or an image database associated with the user (e.g., an online image library associated with the user and / or local images on the user's computing device) to determine images that are relevant to a particular context (e.g., relevant to the user input, the web resource, and / or the generated prompt). Image suggestions may include determining one or more entities, themes, and / or features associated with the web resource and / or the user's input, and then processing a stock image library and / or an image database associated with the user to determine one or more specific images associated with the one or more entities, themes, and / or features associated with the web resource and / or the user's input. For example, if the web resource discusses pasta recipes, a stock image library and / or a user image library may be searched for images depicting pasta, cooking, pasta ingredients, and / or kitchens. Another example may include determining that the text of the generated prompt may be associated with a trip to Mexico, and one or more images from the user's image library may be identified and suggested based on location metadata, feature detection, optical character recognition, and / or other determination techniques that may be used to identify that the one or more images are associated with a trip to Mexico. In some implementations, image suggestions can be based on generating hint embeddings and / or web resource embeddings and then performing an embedding search based on a plurality of image embeddings associated with one or more image databases. Suggestion determination and display can be performed for images, videos, document files, audio, text data, templates, and / or other data.

[0058] Additionally and / or alternatively, the interactive input interface may include a "Help Me Write" feature. The "Help Me Write" feature may be a selectable user interface feature that may provide a generative language model interface for generating text for user-generated content. The "Help Me Write" feature may include a drop-down menu for selecting a specific tone, style, format, length, and / or other attributes for model-generated text. The "HelpMe Write" feature may process user input to adjust and / or change the style, tone, format, language, vocabulary, length, and / or simplicity of the input text. For example, a user may select a tone from a plurality of tone options, enter a text string, and the input interface may provide the text string and the selected tone prompt to a generative language model (e.g., a large language model) to generate a model-generated text response, which may then be used for user-generated content (e.g., link annotations and / or stand-alone content). Alternatively and / or additionally, the input interface may interact with different generative language models associated with different attributes in response to selections of different attribute options. Different generative models can be trained and / or tuned for specific properties.

[0059] The systems and methods of the present disclosure provide multiple technical effects and benefits. As an example, the systems and methods can provide an interactive user interface that can be used to generate prompts and obtain user input data. Specifically, the systems and methods disclosed herein can utilize one or more machine learning models to determine when to request link annotations and generate prompts for requesting information. For example, the generation model can process user data, content data, and / or other context data to determine that an information request action is to be performed. Additionally and / or alternatively, the generation model can generate prompts for requesting information based on user data, content data, and / or other context data. Prompts can be provided to users, user input can be received, and link annotations can be generated and stored.

[0060] Another technical benefit of the disclosed system and method is the ability to utilize user data and content data to determine which users can provide trustworthy information about a particular web resource and / or to determine when to prompt a user to provide information. For example, it may be determined that a user is knowledgeable about a particular topic and / or is a common comment poster for a given type of content. Based on this determination, a user may be prompted to provide a link comment for a given web resource. Alternatively and / or additionally, the subject matter of the content, the type of content, and / or other interactions with the content may be utilized to determine whether a web resource is "appropriate" for commenting. A generative model may be used to generate prompts to provide both user-aware and content-aware prompts.

[0061] The systems and methods disclosed herein solve the problems generated by computing systems obtaining, processing, and sending data from multiple databases from multiple sources. The vast amount of data available to users may provide the possibility of false positives, misleading, and / or lack of verification. Text snippets, titles, and / or example images in the search results interface may provide some details about the content of the web resource; however, information from other users may provide further insights about the subject matter, trustworthiness, and / or expected content, and the further insights may be utilized to reduce instances of users navigating and consulting irrelevant web resources.

[0062] Another example of technical effects and benefits relates to improved computing efficiency of computing systems and improvements in their functionality. For example, the systems and methods disclosed herein can utilize annotation generation to provide an interface that provides information about links that can alleviate the tediousness of search result review by providing user-based verification. The reduced amount of subsequent queries and the reduced amount of page redirects can reduce latency at user devices and can reduce search engine computing costs. With reference now to the accompanying drawings, example embodiments of the present disclosure will be discussed in more detail.

[0063] Figure 1 A block diagram of an example link annotation generation system 10 according to an example embodiment of the present disclosure is depicted. In some implementations, the link annotation generation system 10 is configured to receive and / or obtain context data 12 describing user data, content data, and / or other context data associated with a web page and / or a viewing stance, and as a result of receiving the context data 12, generate, determine, and / or provide a predictive prompt 18 that describes a generated natural language request for information from a user. Thus, in some implementations, the link annotation generation system 10 may include a generation model 16 that is operable to process the context data 12 to generate a predictive prompt 18 that includes a text string describing a question, command, template, and / or suggested comment.

[0064] Specifically, the link annotation generation system 10 can obtain context data 12. The context data 12 may include user data (e.g., data associated with users viewing search result pages, entering search queries, and / or viewing discovery feeds), content data (e.g., data associated with content in web resources 14), and / or other context data (e.g., time, query trends, comment trends, news, etc.). The context data 12 may include user search history data, user browsing history data, user purchase history data, user profile data, user annotation history data, web resource topic tag data, web resource content type tags, other annotations about web resources, and / or other data. The context data 12 may be generated using a personalized machine learning model and / or one or more other machine learning models.

[0065] The contextual data 12 may be obtained based on the web resource 14 being provided for display, previously viewed, and / or associated with a search result in a search results page. The contextual data 12 may be obtained and / or generated based on one or more user interactions, one or more global trends, and / or based on the web resource 14 being associated with a particular type of content (e.g., editorials, tutorials, blogs, news articles, sports score trackers, etc.).

[0066] A generative model 16 (e.g., an autoregressive language model, a diffusion model, and / or one or more other generative models) can process the context data 12 to generate a prediction prompt 18. The generative model 16 can include a language model (e.g., a large language model, a visual language model, and / or other language models), a text-to-image generative model, and / or other generative models. The prediction prompt 18 can include text data, image data, audio data, latent coding data, and / or multimodal data. The prediction prompt 18 can include questions to which a user can respond, a template for drafting a comment, and / or one or more selectable comment options. For example, the prediction prompt 18 can include questions generated based on semantic analysis and / or topic determination for a web resource, can include templates generated based on comments previously generated by a specific user and / or other users, and / or selectable comment options based on previous comments provided for similar web resources. In some implementations, the prediction prompt 18 can describe a request for a specific type of information about the web resource 14. Alternatively and / or additionally, the prediction prompt 18 can describe general information about the web resource 14. Predictive hint 18 may include a novel text string not previously provided by a user and / or related to web resource 14. Predictive hint 18 may include a plurality of predicted characters, words, pixels, signals, and / or structures.

[0067] Prediction prompts 18 may be provided in the input entry interface for display. A user may interact with the input entry interface to generate user generated content 20, which may be sent to a server computing system (e.g., a search engine computing system). The user generated content 20 may include text data, image data, audio data, video data, latent coding data, and / or multimodal data. The user generated content 20 may describe annotations about web resources 14. An annotation may describe comments, opinions, reviews, verifications, and / or indications of quality and / or themes. The user generated content 20 may include annotations displayed in a graphics card having one or more graphics, one or more widgets, one or more links, one or more media content items, and / or a graphics background.

[0068] The link annotation generation system 10 may index the link annotations 22 along with the web resources 14. The index may be utilized to provide user-generated content 20 including the link annotations for display when providing search results for the web resources 14. Alternatively and / or additionally, the user-generated content 20 may be stored in an annotation database for display in an annotation interface when selected by one or more users.

[0069] Figure 2 A block diagram of an example user prompting system 200 is depicted according to an example embodiment of the present disclosure. The user prompting system 200 is similar to Figure 1 The link annotation generating system 10 is different in that the user prompting system 200 further includes an action determining block 230.

[0070] The user prompt system 200 can obtain content data 224 and user data 226. These data can be obtained in response to a search query, a back-off event to a search results page (e.g., returning to a search results page after viewing a web resource), the next search instance, the next instance of a web resource as a search result, and / or based on other triggering events. The content data 224 can describe the content in the web resource, which can include text, images, videos, layouts, audio files, transitions, potentially encoded data, related links / web resources, interaction history for a web resource, and / or other data associated with the web resource and / or other similar web resources. User data 226 may include data describing: user search history (e.g., a log of prior queries obtained from a user), user browsing history (e.g., a log of previously visited web pages and / or platforms), user application history (e.g., a log of prior interactions with applications), user purchase history (e.g., a log of previously obtained products and / or services), user profile (e.g., a user identifier, user preferences, user name, user account, and / or user contacts), annotation history (e.g., a log of previously provided / generated annotations), and / or social media networks and / or activities.

[0071] The content data 224 and / or the user data 226 may be processed using a context determination block 228 to determine context. The context determination block 228 may include one or more machine learning models and / or one or more deterministic functions. The context determination block 228 may generate context data.

[0072] The context data may be processed with an action determination block 230 to determine an input request action to be performed. The action determination block 230 may include one or more machine learning models and / or one or more deterministic functions. The context determination and / or action determination may be performed based on a heuristic approach.

[0073] The input request action may include generating a prompt and providing an input entry interface with the prompt to the user to obtain link annotations for a given web resource. The input request action may be determined based on the likelihood of a user to respond, the trustworthiness of the user, the experience of the user, the knowledge of the user, a difference in association between the user and previous annotation providers, a content gap describing a difference in annotations for a particular web resource versus similar web resources, a review gap describing a difference between interactions with links versus volume of annotations on one or more blogs or social media platforms, a topic of the web resource, a topic of the search, a type of content, an intent of the content, and / or other data.

[0074] Particular content types (e.g., news articles, short stories, movies, skits, blog posts, and / or social media posts) may be determined to be more likely to be interacted with for link annotation generation, and / or may be determined to gain more benefit from link annotations. Additionally and / or alternatively, interactions with web resources on other platforms may be determined. If it is determined that the amount and / or quality of interactions on other platforms meet a threshold difference compared to the current platform, input request actions may be determined more frequently. For example, a threshold for input request actions may be adjusted based on interactions on other platforms. Alternatively and / or additionally, a threshold may be adjusted based on search and / or viewing trends associated with a web resource.

[0075] The input request action may be performed immediately following the determination and / or may be provided as a "nudge" at a later time, which may be a determined time when the likelihood of a response is higher (e.g., when the user is at a particular location (e.g., at home), when the user's calendar is empty, a particular time of day when phone activity increases, and / or at the next user search instance). The "nudge" may be provided via device notification, email, and / or application-based notification.

[0076] The user prompt system 200 can then determine to utilize contextual data to generate a prompt for requesting annotations from the user based on the input request action. The contextual data can include user data 226 (e.g., data associated with a user viewing a search results page, entering a search query, and / or viewing a discovery feed), content data 224 (e.g., data associated with content in a web resource), and / or other contextual data (e.g., time, query trends, comment trends, news, etc.). The contextual data can include user search history data (e.g., a list of previously searched search queries, which can include queries associated with the same topic as the web resource), user browsing history data (e.g., a list of previously viewed web pages, which can include web pages associated with the same topic as the web resource), user purchase history data, user profile data (e.g., a user's name, occupation, education, preferences, etc.), user annotation history data, topic tag data for a web resource, content type tags for a web resource, other annotations about a web resource, and / or other data. The contextual data can be generated using a personalized machine learning model and / or one or more other machine learning models.

[0077] The contextual data may be obtained based on the web resource being provided for display, previously consulted, and / or associated with a search result in a search result page. The contextual data may be obtained and / or generated based on one or more user interactions, one or more global trends, and / or based on the web resource being associated with a particular type of content (e.g., editorials, tutorials, blogs, news articles, sports score trackers, etc.).

[0078] A generative model 216 (e.g., a text generative model, an image generative model, an audio generative model, a video generative model, and / or a multimodal media content item generative model) can process context data to generate a prediction prompt 218. The generative model 216 can include a language model (e.g., a large language model, a visual language model, and / or other language models), a text-to-image generative model, and / or other generative models. The prediction prompt 218 can include text data, image data, audio data, latent coding data, and / or multimodal data. The prediction prompt 218 can include questions to which a user can respond, a template for drafting a comment, and / or one or more selectable comment options. For example, the predictive prompt 218 may include questions generated based on semantic analysis and / or topic determination for a web resource (e.g., for an article about a cold case, the prompt may include "what are your thoughts on the analysis of the cold case?", "who do you think committed the crime?", "was the article understandable and comprehensive with regards to the forensic evidence?", etc.). In some implementations, the predictive prompt 218 may include a template generated based on comments previously generated by a particular user and / or other users (e.g., if a user typically begins their comments with a preposition, the prompt may include a template that begins with a preposition that mimics the style and tone of the previous comments). Additionally and / or alternatively, the predictive prompt 218 may include selectable annotation options based on previous reviews provided for similar web resources (e.g., "the analysis of the case was thorough, understandable, and had a plausible conclusion," "forensic analysis lacked basis in verified science," "the article is more of a fan fiction that an actual article," etc.).In some implementations, the predictive hint 218 may describe a request for a specific type of information about a web resource (e.g., for a biography of a politician, "what were your thoughts on their upbringing?", "please provide your insight on the epilogue," "in my experience as a congressional historian, this biography is accurate / untrustworthy / well-written / poorly-structured," etc.). Alternatively and / or additionally, the predictive hint 218 may describe general information about the web resource. The predictive hint 218 may include a novel text string that has not been previously provided by a user and / or is related to a web resource. The predictive hint 218 may include multiple predicted characters, words, pixels, signals, and / or structures.

[0079] In some implementations, specific terms, details, and / or structure of a search query may be utilized to determine a user's level of experience and / or knowledge about a particular topic. The search query may be included in the context data, and the generation model 216 may generate a predictive hint 218 that reflects the determined level of experience and / or knowledge. Additionally and / or alternatively, a search query chain may be determined utilizing previous search queries to determine search intent. The search intent may then be utilized to generate a predictive hint 218 associated with the search intent.

[0080] In some implementations, the predictive prompt 218 may be different based on the user's propensity to provide link annotations, based on the user's previous annotations, based on the user's credibility, and / or other user data. If the user has never provided a link annotation before and / or has only provided a small number of link annotations before, the predictive prompt 218 can be configured as a general prompt, a multiple-choice prompt, and / or a conversational format. The predictive prompt 218 for experienced users can be generated to provide the user with direct prompts, annotation templates, and / or options based on previous interactions.

[0081] Prediction hints 218 may be provided in the input entry interface for display. A user may interact with the input entry interface to generate user generated content 220, which may be sent to a server computing system (e.g., a search engine computing system). User generated content 220 may include text data, image data, audio data, video data, latently encoded data, and / or multimodal data. User generated content 220 may describe annotations about web resources. Annotations may describe comments, opinions (e.g., “I think the trade discussed in the article was fairbased on the long-term outcomes for both teams.”), critiques (e.g., “the short story lacked proper pacing and direction with the main character having zero character growth”), validations (e.g., “the facts in this article match those of other reputable sources”), and / or indications of quality and / or theme (e.g., “such a well-written play on the perils of love in war stricken towns”). User-generated content 220 may include annotations displayed in a graphic card having one or more graphics, one or more widgets, one or more links, one or more media content items, and / or a graphic background. For example, the text of a link annotation may be provided as stylized text with color to juxtapose with the model-generated image used as a background.

[0082] In some implementations, the generation model 216 can process the user-generated content 220 and can generate a follow-up prompt. The follow-up prompt can request additional information and / or can provide options for further customization.

[0083] The user prompt system 200 can store the annotations 222 with the web resource. The index can be used to provide user-generated content 220 including the link annotations for display when providing search results for the web resource. Alternatively and / or additionally, the user-generated content 220 can be stored in the annotation database so as to be displayed in the annotation interface when selected by one or more users.

[0084] For example, a particular user and / or other users may enter a search query. The search engine system may determine that a web resource is responsive to the search query. Search results associated with the web resource may be provided in a search result interface (e.g., a graphics card) having the web resource's title, media clips, and data describing link annotations.

[0085] Figure 3 Depicted is a flowchart of an example method performed in accordance with an example embodiment of the present disclosure. Figure 3 For the purpose of illustration and discussion, the steps are depicted to be performed in a specific order, but the method of the present disclosure is not limited to the specific illustrated order or arrangement. The steps of the method 300 may be omitted, rearranged, combined and / or adapted in various ways without departing from the scope of the present disclosure.

[0086] At 302, a computing system may obtain content data. The content data may be associated with a web resource. The content data may describe the content of the web resource, which may include text data, image data, video data, audio data, potentially encoded data, and / or multimodal data. The content data may include data describing the subject of the web resource, the type of content, other annotations received from other users, web resource metadata, the author of the web resource, and / or entities associated with the web resource. The content data may include content tags, the entirety of the web resource content, a summary of the content, media segments, and / or content embeddings.

[0087] At 304, the computing system may process the content data with a generative model to generate a predictive prompt. The prompt may include a predicted text string associated with commenting on a web resource. The generative model may include an autoregressive language model. In some implementations, the generative model may be prompted to generate questions describing requests for information about web resources. The generative model may include a transformer model. The generative model may have been trained, configured, and / or prompted to perform semantic understanding on web resources, and then generate prompts (e.g., questions) based on the semantic understanding. The predictive prompt may include a plurality of predicted characters, which may be determined specifically based on the content data. The predictive prompt may inquire about the quality of the web resource. The predictive prompt may inquire about opinions and / or reviews of the web resource.

[0088] In some implementations, the computing system may obtain user data. The user data may be associated with a particular user. Processing the content data with the generative model to generate the predictive prompt may include: processing the content data and the user data with the generative model. The user data may include user search history data, user browsing history data, the user's social network, user preferences, user profile information, the user's location, the user's purchase history, and / or user contacts. The generative model may generate the predictive prompt based on the information that the particular user previously searched for and was associated with the theme of the web resource. Alternatively and / or additionally, the generative model may generate the predictive prompt based on the particular user's previous viewing of other web resources that include information associated with the theme of the web resource. The generative model may determine that the particular user is associated with a particular theme, a particular type of content, a particular opinion, and / or a particular comment context, and may generate the predictive prompt based on the determination.

[0089] At 306, the computing system may provide a prediction prompt for display with an input prompt interface. The input prompt interface may be configured to receive input. The input prompt interface may include multiple selectable user interface elements. In some implementations, the input prompt interface may include an input entry box for receiving input from a user. The input prompt interface may include an upload element for uploading media content items (e.g., documents, images, text, videos, audio files, etc.). In some implementations, the input prompt interface may include an interface for providing input to the generation model for the user to generate model-generated annotations based on user input. Alternatively and / or additionally, multiple selectable user interface elements may include one or more selectable templates for generating user-generated content (e.g., user-generated annotations). One or more templates may be generated based on content items previously generated by a specific user (e.g., previously generated annotations).

[0090] At 308, the computing system may obtain comment input data from the user computing system via an input prompt interface. The comment input data may include text data, image data, audio data, potentially encoded data, and / or multimodal data. The comment input data may include one or more selections, one or more text strings, and / or one or more uploaded files. The comment input data may include user-generated comments on the web resource. The user-generated comments may include comments on the quality of the web resource, the subject of the web resource, and / or other aspects of the web resource.

[0091] At 310, the computing system may store data associated with the comment input data with data associated with the web resource. Storing data associated with the comment input data with data associated with the web resource may include generating a web resource annotation and storing the web resource annotation with a plurality of other web resource annotations associated with the web resource. The comment input data may be indexed in association with the web resource and may be stored in a database to be provided with search results for the web resource. The data associated with the comment input data may be stored in a searchable database to be provided for display in response to the web resource being determined and / or provided as a search result.

[0092] In some implementations, the computing system can provide the web resource annotation and the plurality of other web resource annotations in an annotation interface that provides the web resource annotation and the plurality of other web resource annotations in a plurality of graphics cards.

[0093] Additionally and / or alternatively, the computing system may generate a graphics card based on the user data, the content data, and the comment input data. The graphics card may include a user profile identifier for a particular user and data associated with the comment input data. The computing system may then store the graphics card. The graphics card may include a graphic background generated based on the comment input data using an image generation model.

[0094] In some implementations, the computing system can obtain a search query, determine that a web resource is associated with the search query, and provide specific search results for display. The specific search results can include a link to the web resource, a title of the web resource, and data associated with the comment input data.

[0095] Figure 4 Depicted are diagrams of example prompts according to example embodiments of the present disclosure. Specifically, Figure 4 An example input entry interface 402 is depicted in FIG. The example input entry interface 402 can be provided in response to determining that an input request action is to be performed and a prediction prompt is generated. The input entry interface 402 can include a link to a web resource 404 and / or a reference to the web resource, a composition panel 406 for displaying input provided during user-generated content generation, one or more selectable prediction prompts 408, and / or one or more user interface elements for providing audio input and / or multimedia input (e.g., images).

[0096] One or more selectable predictive hints 408 may include predictive hints generated by processing the content of web resource 404 with a generative language model. Alternatively and / or additionally, one or more selectable predictive hints 408 may include predictive hints generated by processing annotations associated with articles similar to the articles provided by web resource 404 with a generative language model.

[0097] The suggested prediction prompts may be provided in a variety of formats. Additionally and / or alternatively, the number and / or length of the prediction prompts may vary based on the content, user, and / or other contextual data. For example, three options 412 may be provided or ten options 414 may be provided. The annotation options may include a plurality of selectable prompts that may be selected by the user as the user's link annotation and / or as part of the user's link annotation.

[0098] The input entry interface 402 may be used to receive text input (e.g., via a graphical keyboard interface), audio input (e.g., via one or more microphones), selections (e.g., selections of user interface elements associated with a predictive prompt annotation option), and / or media content item input (e.g., image uploads). The received input may be provided for display in a preview window of the composition panel 406, and the received input may then be utilized to generate a user-generated content item, which may include a link annotation.

[0099] Figure 5A A diagram of an example annotation interface with topic prompts according to an example embodiment of the present disclosure is depicted. The example annotation interface can be used to learn more about web resources 502, read other users' thoughts on web resources 502, and / or generate and provide new annotations about web resources 502. Specifically, Figure 5A Depict a link to a web resource 502, a previously provided note 504, one or more prompts 506 for requesting information from a user, and an approval user interface element 508. The link to the web resource 502 may include a thumbnail, URL, and title. The previously provided note 504 may include an image and / or text provided by another user. The previously provided note 504 may be provided with interactive data, including approval, comments, and / or likes. One or more prompts 506 for requesting information from a user may include a discussion topic to which a user responds in the user's note. The approval user interface element 508 may be used to interact with the web resource 502 and / or the previously provided note 504.

[0100] Figure 5BAn illustration of an example annotation interface with similar article prompts according to an example embodiment of the present disclosure is depicted. The example annotation interface can be used to learn more about web resource 502, read other users' thoughts on web resource 502 (e.g., opinions on the subject matter of the web resource and / or the quality of the web resource), and / or generate and provide new annotations about web resource 502 (e.g., users can provide their own details and / or thoughts about the web resource). Specifically, Figure 5B Depict a link to web resource 502, a previously provided note 504, a user interface element 510 for commenting on an article similar to web resource 502, and a suggested article 512. The link to web resource 502 may include a thumbnail, URL, and title. The previously provided note 504 may include an image and / or text provided by another user. The previously provided note 504 may be provided with interactive data, including recognition, comments, and / or likes. The user interface element 510 for commenting on an article similar to web resource 502 may be selected to open an input entry interface for providing link notes about one or more similar web resources (which may include suggested article 512).

[0101] Figure 5C An illustration of an example predictive prompt according to an example embodiment of the present disclosure is depicted. Specifically, content data associated with a web resource and / or user data associated with a particular user being prompted may be processed with a generative model to generate one or more predictive prompts. Figure 5C Depicted are example question prompts 514 and starting point prompts 516. Question prompts 514 may include questions for asking a user to provide information about a particular topic and / or subtopic. Starting point prompts 516 may include an introductory sentence for the user to construct the user's link annotation based on. When provided in a search result interface and / or an annotation interface, user-generated content including a link annotation may include a start prompt.

[0102] FIG. 6A to FIG. 6C An illustration of an example link annotation entry point according to an example embodiment of the present disclosure is depicted. An entry point to a link annotation generation interface may be provided in a plurality of different interfaces and / or media. Specifically, Fig. 6A An example search results interface is depicted with example search results 602. Context data associated with a search instance (eg, user data, search query, and / or search results) can be processed to determine that a link annotation hint is to be provided. Fig. 6A A first example prompt 604 is depicted that is provided in response to determining that the user has previously searched for the topic. Additionally, Fig. 6ADepicted is a second example prompt 606 provided in response to determining an interaction trend and / or determining that a particular web resource is a type of content that is frequently interacted with for annotation generation.

[0103] Figure 6B Depicted are three different entry points for linking annotation prompts and generation. The first entry point 608 may include a drop-down menu of a browser application. The user may select an option drop-down menu and select an annotation generation option from the drop-down menu. The second entry point 610 may be included in the search results interface. The user may select an option in the search results interface to comment on previously viewed content. The third entry point 612 may include an overlay user interface element that may be provided in a viewing window of the content of a web resource. The overlay user interface element may be provided by an operating system of a web resource, a browser, and / or a user computing device.

[0104] Figure 6C A search review entry point is depicted. Specifically, a widget and / or graphical pane may be provided to the user to provide a review of previous searches, which may include selecting a drop-down list element 614 to select a web resource from a viewed content list 616 to indicate which web resource was helpful to the user's particular search. The selection may be used to generate a link annotation, re-rank the web resource, and / or navigate to a link annotation generation interface.

[0105] Figure 7 Depicted is a flowchart of an example method performed in accordance with an example embodiment of the present disclosure. Figure 7 The steps are depicted in a specific order for the purpose of illustration and discussion, but the method of the present disclosure is not limited to the specific illustrated order or arrangement. The steps of method 700 may be omitted, rearranged, combined and / or adapted in various ways without departing from the scope of the present disclosure.

[0106] At 702, the computing system may obtain contextual data. The contextual data may be associated with a particular content display instance. The particular content display instance may include a particular user viewing a particular content item. In some implementations, the contextual data may be associated with the type of content being provided for display (e.g., an article, an academic paper, a blog post, an encyclopedia entry, a video, a media content library, and / or other types of content). Additionally and / or alternatively, the contextual data may be associated with a particular user associated with the particular content display instance. The contextual data may include search history data, browsing history data, user profile data, purchase history data, social network data, and / or other user data.

[0107] At 704, the computing system may determine an input request action based on the context data. The input request action may include providing an input entry interface to the user to obtain user input. The content being provided for display may be associated with a specific web resource. The context data may be associated with interactive data (e.g., linked posts, comments, forwarding, likes, and / or mentions) for links to specific web resources on multiple social networking platforms. In some implementations, the input request action may be determined based on the interactive data. The context data may include user data and content data. Additionally and / or alternatively, the input request operation may be determined based on a topic associated with the content being provided for display, based on user data determining that a specific user is aware of one of a plurality of topics.

[0108] At 706, the computing system may process the context data with the generative language model to generate a predictive hint. The predictive hint may include a natural language request for information generated based on the context data. The context data may include previous annotations generated by a particular user. The predictive hint may include a structure based on a previous structure of previous annotations. In some implementations, the generative model may process the content data to generate a predictive hint based on the content of a particular web resource.

[0109] At 708, the computing system may provide a predictive prompt in an input entry interface. The predictive prompt may be provided adjacent to an input entry box for receiving and displaying input text and / or images. The input entry interface may include a panel adjacent to search results for a particular web resource. Alternatively and / or additionally, the input entry interface may be provided in the form of a pop-up interface, and / or may be redirected based on one or more inputs.

[0110] At 710, the computing system may obtain user-generated content via an input entry interface. The user-generated content may include text data, image data, video data, audio data, latently encoded data, statistical data, and / or multimodal data. The user-generated content may be obtained via an upload interface and / or an input entry box.

[0111] At 712, the computing system may generate a link annotation based on the user-generated content. In some implementations, the computing system may generate a graphics card including the link annotation. The graphics card may include a graphic background that may be selected by the user and / or may be automatically generated. The graphic background may be generated based on the content of the web resource, the content of the link annotation, and / or the type of annotation. The graphics card and / or the link annotation may be stored to be provided with data associated with the web resource. The link annotation may be generated to be provided in the search results interface for display in response to a particular content item being determined as a search result.

[0112] Figure 8Depicted is a flowchart of an example method performed in accordance with an example embodiment of the present disclosure. Figure 8 The steps performed in a specific order are depicted for the purpose of illustration and discussion, but the method of the present disclosure is not limited to the specific illustrated order or arrangement. The steps of method 800 may be omitted, rearranged, combined and / or adapted in various ways without departing from the scope of the present disclosure.

[0113] At 802, a computing system may obtain a first search query at a first time and determine that a web resource is responsive to the first search query. The first search query may include a text query, an image query, an audio query, an embedded query, and / or a multimodal query. The web resource may be identified via a search engine that may perform a keyword search, an embedding-based search, and / or other search techniques. The web resource may be determined to be responsive to a subject, question, and / or intent of the first search query.

[0114] At 804, the computing system can obtain content data and process the content data with a generative model to generate a prediction prompt. The content data can be associated with a web resource. The prompt can include a predicted text string associated with commenting on the web resource. The content data can describe the overall content of the web resource, media segments, abstracts of the content, content tags, metadata, and / or content of previously provided link annotations associated with the web resource. The generative model can process the content data to determine a theme, viewpoint, intent, topic, structure, target audience, content type, and / or other content details. Then, a prediction prompt can be generated based on this determination.

[0115] At 806, the computing system may provide a prediction prompt for display within an input prompt interface, and obtain comment input data from the user computing system via the input prompt interface. The input prompt interface may include an input entry box. In some implementations, the input prompt interface may include multiple user interface elements for drafting content (e.g., comments). The comment input data may include user-generated content. In some implementations, the comment input data may include multimodal data. The multimodal data may include text data and image data.

[0116] At 808, the computing system may store the user-generated content. The user-generated content may be indexed along with links to web resources. Alternatively and / or additionally, the web resources may be indexed along with the user-generated content. The user-generated content may be stored along with other user annotations associated with a particular web resource and / or a particular user.

[0117] At 810, the computing system may obtain a second search query at a second time and determine that the web resource is responsive to the second search query. The second time may be different from the first time. In some implementations, the first search query and the second search query may be different. The second search query may include a text query, an image query, an audio query, an embedded query, and / or a multimodal query. The web resource may be identified via a search engine that may perform a keyword search, an embedded-based search, and / or other search techniques. The web resource may be determined to be responsive to a topic, question, and / or intent of the second search query.

[0118] At 812, the computing system can provide the user-generated content in a search results interface with data describing the web resource. The user-generated content can be provided along with a link, title, and text snippet of the web resource.

[0119] FIG. 9A to FIG. 9G A diagram of an example graphics card interface is depicted in accordance with an example embodiment of the present disclosure.The systems and methods disclosed herein may be utilized to generate a graphics card for user-generated content that may include linked annotations and / or stand-alone content.

[0120] Fig. 9A Two example graphic cards are depicted. The depicted graphic cards may include a user profile identifier 902 (e.g., a user profile image and name), a body of a card 904 (e.g., a link annotation in the form of stylized text overlaid on a graphic background), a widget interface element 906 (e.g., a selectable user interface element for redirecting to a web resource and / or additional content), and / or interactive information 908 (e.g., likes, comments, and / or saves for the graphic card). The body of the card 904 may be configured by a particular user, and / or may be automatically generated based on the link annotation, the web resource, and / or user preferences. The widget interface element 906 may include a link to a web page, a link to an image gallery, a link to a video, a selectable element for opening a pop-up interface, additional annotations, and / or other data.

[0121] Fig. 9B Depicted are example multi-page user-generated content and example video user-generated content. The multi-page user-generated content may include multiple graphics cards that may be cycled through to display the user-generated content. The video user-generated content may include a video with graphics and / or text superimposed on it.

[0122] Widget interface elements 906 may include links to web resources, links to one or more other web resources, video elements that can be selected to provide a video for display, media content display elements for providing media content (e.g., videos, images, audio files, and / or other media) for display, reviews for the web resource, links to other annotations, structured content items (e.g., structured recipes and / or structured calculators), lists (e.g., ingredient lists), map location cards (e.g., a map associated with a web resource and / or a link to a web application), knowledge panels, and / or links to shopping interfaces.

[0123] Fig. 9C Example interactions with the graphics card are depicted. For example, a user profile identifier 902 can be selected to peek into a user's profile 910. Additionally and / or alternatively, a video widget element 912 can be selected to expand a video for playback and / or to navigate to a video player interface. The graphics card can be selected to minimize the plugin 914. A knowledge panel widget element 916 can be selected to expand a knowledge panel to provide additional information for display. Interactive element 918 can be selected to like, comment, save, and / or share user-generated content.

[0124] Fig.9D Depicted are the prominence levels of widget interface elements (e.g., plug-in elements). Specifically, the ingredients and instructions for the depicted smoothie can be provided in medium prominence interface element 922 (e.g., detailed view state), low prominence interface element 924 (e.g., collapsed state), and / or high prominence graphical panel 926 (e.g., expanded state). In some implementations, a user can interact with the widget interface element to transition between detail levels and / or sizes.

[0125] Fig.9E Depicted is an annotation search result in search result interface 930. An annotation search result can be provided in a separate tab, provided adjacent to other search results, and / or provided in a category-specific panel. An annotation search result 932 can be selected to navigate to an immersive viewer 936 that displays an expanded view of the user-generated content.

[0126] Fig.9F Different graphics card displays and / or annotation interface displays are depicted. The graphics card may be displayed in a vertically scrolling single width format 940, an offset vertically scrolling double width format 942, a horizontally scrolling carousel interface within other search result formats 944, and / or an aligned vertically scrolling double width format 946. The format may be based on theme, type of interface, user preferences, and / or context.

[0127] Figure 9GDifferent customization options can be depicted. For example, a graphics card customization interface can be provided to generate a graphics card, which can include editing text 952, editing layout 954, editing image 956 and / or other customization options. Specifically, the interactive interface can include multiple options (and / or features) for content generation. Multiple interface features can include text, images, audio, video, templates and / or other input options. Multiple interface features can include content suggestions, template suggestions and / or one or more generation model interfaces for generating model-assisted generation (e.g., a large language model for rewriting text and / or actively generating text; an image generation model for generating novel images based on web resources, user input and / or generated prompts; an audio generation model for generating narration, songs and / or other audio; and / or a graphics card generation model for processing web resources, generated prompts and / or user input to generate a graphics card that can be suggested to users for link annotations and / or independent content). Multiple interface features can include customization options for customizing layouts, fonts, interface element sizes, images, text, transitions, tones, shadows and / or other user-generated content features. The plurality of interface features may include an option to add an action user interface element to a graphics card of user-generated content. The action user interface element may include selectable options for performing one or more actions (e.g., an API call, navigating to a different application, searching, generating a content item using a generative model, etc.).

[0128] The systems and methods disclosed herein may include image suggestions and / or image generation for generating a graphics card. For example, the systems and methods may determine that images from a database (e.g., a server database, a local database, and / or a user image gallery) are associated with web resources, prompts, and / or link annotations. These images may then be provided as suggestions for use in the graphics card. Alternatively and / or additionally, the systems and methods may provide an image generation model (e.g., a text-to-image generation model) interface to generate images to be included in the graphics card. For example, an image generation model interface may be provided to a user, the user may provide prompts to the image generation model, and the image generation model may generate a model-generated image that may then be used in the graphics card.

[0129] In some implementations, one or more machine learning models may be used to fact-check web resources and / or link annotations. The one or more machine learning models may include one or more generative models that may utilize an application programming interface to make API calls to obtain information and / or interact with other applications.

[0130] In some implementations, a generative model can be utilized to generate one or more model-generated link annotations, which can be indexed together with web resources to provide link annotation examples and / or provide semantic understanding annotations. Alternatively and / or additionally, a generative model can be utilized to rewrite and / or suggest link annotations and / or independent content. An interactive user interface may include an interface for interacting with a generative model to generate content (e.g., text, images, and / or other data). An interactive user interface may include options for selecting tone, style, format, vocabulary, genre, and / or other attributes to be used to adjust a generative model to generate content with specific attributes. For example, an interactive user interface may be configured to generate prompts for a generative model based on user input, link annotation prompts, and / or web resources.

[0131] The search results interface and / or the discovery interface may provide statistics regarding the volume of a particular search, the volume of web resource selections, and / or trends in links and / or search query interactions.

[0132] In some implementations, systems and methods may include training and / or utilizing one or more contribution propensity models. The contribution propensity model may learn and / or determine user credibility (e.g., the user's relevant experience, expertise, and / or trustworthiness) for a particular user and / or a particular set of users. Additionally and / or alternatively, the contribution propensity model may learn and / or determine a propensity to provide link annotations.

[0133] The contribution tendency model can be trained to detect the likelihood of contribution, credibility, usefulness of annotations, and / or other attributes associated with users, web resources, and / or contexts. The contribution tendency model can be trained based on labeled data sets, based on unlabeled data sets, and / or based on hybrid data sets. In some implementations, the contribution tendency model can be trained based on interaction data for learning contribution prediction tasks, can be trained based on the output of the verification model for credibility determination tasks, and / or can be trained based on click-through rates for usefulness determination tasks.

[0134] Fig.10 An illustration of an example card generation interface 1000 is depicted according to an example embodiment of the present disclosure. Specifically, a user can select an option to generate a graphical card for a link annotation. Based on the selection, a template 1002 can be selected and provided for display. A particular template can be selected based on a web resource associated with the link annotation (e.g., based on the content of the web resource), based on a user interaction history, based on a query history, based on user profile data, and / or based on other data.

[0135] The card generation interface 1000 may include a pull-up menu 1004 associated with a plurality of suggested prompts for link annotation generation, which may include a theme. The user may pull up the menu to provide an expanded view 1006 of suggested prompts. The expanded view 1006 may include a plurality of selectable suggested prompts, which may be selected to generate text, images, and / or layouts to be inserted into a graphics card. For example, a "How to Water Your Monstera Like a Pro" suggestion may be selected. Content items associated with the selected prompt suggestion may be inserted into a graphics card, which may be transformed to an editing interface 1008. The editing interface 1008 may include options for editing text, style, layout, font, color, and / or other edits.

[0136] Fig.11 Depicted is an illustration of an example content item generation interface 1100 according to an example embodiment of the present disclosure. Specifically, while modifying a graphic card template, an input drafting interface may be utilized to generate one or more content items.

[0137] For example, a user can select an option to open content item generation interface 1100. At 1102, a user can select one or more attributes from a drop-down menu. One or more attributes can be associated with the attribute of the requested content item being generated. One or more attributes can be associated with the tone and / or style of the content. At 1104, a user can generate and / or provide text input. The text input can be associated with a theme, intention, information and / or other prompt details. One or more attributes and text input can be processed with a generation model to generate the content item generated by the model. The content item generated by the model can have one or more attributes, and can be for the theme, intention, information and / or other prompt details of the text input.

[0138] At 1106, the model-generated content item can be provided below the text input for display, and the model-generated content item can be provided with a plurality of options. The plurality of options can include editing one or more attributes, editing the text input, reprocessing the data, saving the model-generated content item, exiting the interface, inserting the model-generated content item into the graphics card, and / or other options. At 1108, a modified graphics card can be provided for display, wherein the model-generated content item is inserted into the graphics card based on user selection. The user can then edit the layout, size, color, font, and / or orientation of the model-generated content item and / or other content of the graphics card.

[0139] Fig.12An illustration of an example image suggestion interface 1200 is depicted according to an example embodiment of the present disclosure. Specifically, the image suggestion interface 1200 can obtain card data, context data, and / or input data. The card data, context data, and / or input data can then be processed to determine one or more images (and / or other media content items) to be provided as suggestions for insertion into a graphics card.

[0140] For example, at 1202, a graphics card is provided for display, and the graphics card has an option to insert additional text, stickers and / or images. Then, the user can choose to add an image option. At 1204, an image selection interface can be provided for display, and the image selection interface can include a default image, a camera album image and / or an image suggestion, which is based on the text of the graphics card, the content of the web resource associated with the link annotation, the user history and / or other data. For example, it can be determined that these images are related to the text of the graphics card based on determining that multiple images from the user's image gallery are associated with the location (e.g., Mexico) referenced in the text of the graphics card. At 1206, the identified image can be provided for display to select. The user can select a specific image from the identified image, and the specific image can be processed and inserted into the graphics card. At 1208, the selected image can be cut and inserted into the graphics card for display.

[0141] Fig.13 Depicted is a flowchart of an example method performed in accordance with an example embodiment of the present disclosure. Fig.13 For the purpose of illustration and discussion, the steps are depicted to be performed in a specific order, but the method of the present disclosure is not limited to the order or arrangement specifically described. The steps of method 1300 may be omitted, rearranged, combined and / or adapted in various ways without departing from the scope of the present disclosure.

[0142] At 1302, the computing system may obtain card data. The card data may describe the content in the graphics card. The content may be associated with one or more themes. The graphics card may be associated with a link annotation. The link annotation may include user-generated content tagged to a specific web resource. The graphics card may include a background, one or more images, one or more text strings, and / or one or more user interface elements. The background may include a single color, multiple colors, images, and / or other data. One or more user interface elements may include selectable widgets for providing additional information for display and / or for performing one or more actions. The content may include text data, image data, video data, potentially encoded data, multimodal data, and / or other data.

[0143] At 1304, the computing system may process the card data to determine one or more entity tags associated with the content. The one or more entity tags may be associated with one or more topics. The card data may be processed with one or more machine learning models (e.g., generative models, classification models, and / or other models) to generate the entity tags. The entity tags may be associated with one or more objects, one or more companies, one or more locations, one or more individuals, one or more structures, and / or other entities.

[0144] At 1306, the computing system may access a media content item database to obtain one or more media content items. The one or more media content items may be obtained based on determining that the one or more media content items are associated with one or more entity tags associated with the content. The media content item database may include a user-specific database. In some implementations, the user-specific database may be associated with a specific user. The specific user may have generated at least a portion of the content. The user-specific database may include an image gallery associated with the specific user. The image gallery may be stored on a server computing system associated with a specific content item storage platform. Alternatively and / or additionally, the user-specific database may include a local storage database of the user computing device. The media content item database may include multiple media content items. Multiple content items may have been preprocessed to generate multiple corresponding metadata sets. Determining that one or more media content items are associated with one or more entity tags associated with the content may include: determining whether the one or more media content items include features associated with the entity tags. Features may be determined based on metadata, image processing, and / or other techniques. The one or more media content items may include one or more images, one or more videos, one or more animations, one or more audio files, and / or one or more other content items.

[0145] At 1308, the computing system may provide one or more media content items for display. The one or more media content items may be provided in an interactive user interface. The one or more media content items may be selected to be inserted into a graphics card. The interactive user interface may provide multiple media content items for display, and the multiple media content items may include media content items associated with the user, web media content items, and / or other media content items.

[0146] In some implementations, a computing system may obtain an input selection associated with one or more media content items and generate an enhanced graphics card. The enhanced graphics card may include at least a portion of the content of the graphics card and at least a portion of the one or more media content items. The computing system may then provide the enhanced graphics card for display.

[0147] Alternatively and / or additionally, the computing system may obtain an enhancement input. The adjustment input may be associated with a request to enhance the enhanced graphics card. The computing system may generate an updated graphics card based on the adjustment input. The updated graphics card may include an enhanced graphics card that has undergone one or more adjustments. The computing system may then provide the updated graphics card for display. The one or more adjustments may include at least one of a layout change of the enhanced graphics card, a cropping change of one or more media content items, a size change of one or more content items, a color change, or a template change.

[0148] Fig.14 Depicted is a flowchart of an example method performed in accordance with an example embodiment of the present disclosure. Fig.14 For the purpose of illustration and discussion, the steps are depicted to be performed in a specific order, but the method of the present disclosure is not limited to the order or arrangement shown in particular. The steps of method 1400 may be omitted, rearranged, combined and / or adapted in various ways without departing from the scope of the present disclosure.

[0149] At 1402, the computing system may provide an input drafting interface for display. The input drafting interface may include a graphical user interface that includes multiple attribute options and a text input box. The multiple attribute options may be associated with multiple candidate attributes for content item generation. The multiple candidate attributes may include tone, style, length, content type, and / or other details. The input drafting interface may include a preview window for viewing the current state of a graphic card. The graphic card may be associated with a link annotation. In some implementations, the graphic card may include a card template that may have been modified based on one or more user inputs. For example, a user may have added images, text, audio, video, widgets, and / or other data.

[0150] At 1404, the computing system may obtain a selection of a specific attribute option from a plurality of attribute options via an input drafting interface. The specific attribute option may be associated with a specific candidate attribute. In some implementations, the plurality of candidate attributes may include a plurality of different styles. The plurality of different styles may be associated with at least one of a plurality of different artistic styles or a plurality of different writing styles. Alternatively and / or additionally, the plurality of candidate attributes may include a plurality of different tones. The plurality of different tones may be associated with at least one of a plurality of different emotions and / or a plurality of different rhythm types. The specific candidate attribute may include a tone and / or style requested for generating a content item. The selection may be obtained based on a selection of a specific attribute option from a drop-down menu that provides a plurality of attribute options for display.

[0151] At 1406, the computing system may obtain text input via a text input box of the input drafting interface. The text input may be associated with a hint map generated by the content item. In some implementations, the text input may be automatically filled based on the content of the graphics card, based on the user context, and / or based on the hint suggestion.

[0152] At 1408, the computing system may process the specific attribute options and text input with the generative model to generate model-generated content items. The model-generated content items may include specific candidate attributes. The model-generated content items may be associated with a hint diagram. The model-generated content items may include text data, image data, audio data, multimodal data, and / or other data. In some implementations, the generative model may be obtained from a generative model database based on the selection of a specific attribute option. For example, the generative model database may store a plurality of different generative models associated with a plurality of candidate attributes. Each generative model in a plurality of different generative models may be configured, trained, and / or tuned to generate content items associated with the corresponding candidate attributes. Alternatively and / or additionally, the generative model may be a general generative model trained for a plurality of content generation tasks. Additionally and / or alternatively, a specific attribute soft prompt may be obtained based on the selection of a specific attribute option. The specific attribute soft prompt may include a learned parameter set. The learned parameter set may be processed with the generative model to generate model-generated content items.

[0153] At 1410, the computing system may provide the model-generated content item for display via the input drafting interface. Providing the model-generated content item for display may include providing an option to insert the model-generated content item into a graphics card. The input drafting interface may include a plurality of post-processing editing options, which may include options to change size, color, font, cropping, resolution, saturation, tinting, and / or other details.

[0154] In some implementations, the computing system may obtain an input selection via an input drafting interface and generate an enhanced graphics card based on the input selection. The enhanced graphics card may include a graphics card enhanced to include a content item generated by the model. The computing system may then provide the enhanced graphics card for display.

[0155] Fig.15A A block diagram of an example computing system 100 that performs link annotation hinting according to an example embodiment of the present disclosure is depicted. The system 100 includes a user computing system 102, a server computing system 130, and / or a third computing system 150 that are communicatively coupled via a network 180.

[0156] The user computing system 102 may include any type of computing device, such as, for example, a personal computing device (e.g., a laptop or desktop computer), a mobile computing device (e.g., a smartphone or tablet computer), a game console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.

[0157] The user computing system 102 includes one or more processors 112 and a memory 114. The one or more processors 112 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be one processor or a plurality of processors operatively connected. The memory 114 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 114 may store data 116 and instructions 118 that are executed by the processor 112 to cause the user computing system 102 to perform operations.

[0158] In some implementations, the user computing system 102 may store or include one or more machine learning models 120. For example, the machine learning model 120 may be or may otherwise include various machine learning models, such as a neural network (e.g., a deep neural network) or other types of machine learning models, including nonlinear models and / or linear models. The neural network may include a feedforward neural network, a recurrent neural network (e.g., a long short-term memory recurrent neural network), a convolutional neural network, or other forms of neural networks.

[0159] In some implementations, one or more machine learning models 120 may be received from the server computing system 130 via the network 180, stored in the user computing device memory 114, and then used or otherwise implemented by the one or more processors 112. In some implementations, the user computing system 102 may implement multiple parallel instances of a single machine learning model 120 (e.g., to perform parallel machine learning model processing across multiple instances of input data and / or detected features).

[0160] More specifically, the one or more machine learning models 120 may include one or more detection models, one or more classification models, one or more segmentation models, one or more enhancement models, one or more generation models, one or more natural language processing models, one or more optical character recognition models, and / or one or more other machine learning models. The one or more machine learning models 120 may include one or more Transformer models. The one or more machine learning models 120 may include one or more neural radiation field models, one or more diffusion models, and / or one or more autoregressive language models.

[0161] One or more machine learning models 120 can be used to detect one or more object features. The detected object features can be classified and / or embedded. The classification and / or embedding can then be used to perform a search to determine one or more search results. Alternatively and / or additionally, one or more detected features can be used to determine that an indicator (e.g., a user interface element indicating the detected feature) is to be provided to indicate that the feature has been detected. The user can then select the indicator to perform feature classification, embedding, and / or searching. In some implementations, the classification, embedding, and / or searching can be performed before selecting the indicator.

[0162] In some implementations, the one or more machine learning models 120 may process image data, text data, audio data, and / or potentially coded data to generate output data, which may include image data, text data, audio data, and / or potentially coded data. The one or more machine learning models 120 may perform optical character recognition, natural language processing, image classification, object classification, text classification, audio classification, context determination, action prediction, image correction, image enhancement, text enhancement, sentiment analysis, object detection, error detection, restoration, video stabilization, audio correction, audio enhancement, and / or data segmentation (e.g., mask-based segmentation).

[0163] Additionally or alternatively, one or more machine learning models 140 may be included in or otherwise stored and implemented by a server computing system 130 that communicates with the user computing system 102 according to a client-server relationship. For example, the machine learning model 140 may be implemented by the server computing system 130 as part of a web service (e.g., a viewfinder service, a visual search service, an image processing service, an environmental computing service, and / or an overlay application service). Thus, one or more models 120 may be stored and implemented at the user computing system 102, and / or one or more models 140 may be stored and implemented at the server computing system 130.

[0164] The user computing system 102 may also include one or more user input components 122 that receive user input. For example, the user input component 122 may be a touch-sensitive component (e.g., a touch-sensitive display screen or a touchpad) that is sensitive to the touch of a user input object (e.g., a finger or a stylus). The touch-sensitive component may be used to implement a virtual keyboard. Other example user input components include a microphone, a traditional keyboard, or other devices by which a user can provide user input.

[0165] In some implementations, the user computing system may store and / or provide one or more user interfaces 124, which may be associated with one or more applications. One or more user interfaces 124 may be configured to receive input and / or provide data for display (e.g., image data, text data, audio data, one or more user interface elements, augmented reality experience, virtual reality experience, and / or other data for display). The user interface 124 may be associated with one or more other computing systems (e.g., server computing system 130 and / or third-party computing system 150). The user interface 124 may include a viewfinder interface, a search interface, a generative model interface, a social media interface, and / or a media content gallery interface.

[0166] The user computing system 102 may include one or more sensors 126 and / or receive data from the one or more sensors. The one or more sensors 126 may be housed in a housing assembly that houses one or more processors 112, memory 114, and / or one or more hardware components that may store one or more software packages and / or cause the execution of one or more software packages. The one or more sensors 126 may include one or more image sensors (e.g., cameras), one or more lidar sensors, one or more audio sensors (e.g., microphones), one or more inertial sensors (e.g., inertial measurement units), one or more biosensors (e.g., heart rate sensors, pulse sensors, retinal sensors, and / or fingerprint sensors), one or more infrared sensors, one or more position sensors (e.g., GPS), one or more touch sensors (e.g., conductive touch sensors and / or mechanical touch sensors), and / or one or more other sensors. One or more sensors may be used to obtain data associated with a user environment (e.g., an image of the user environment, a record of the environment, and / or the user's location).

[0167] The user computing system 102 may include a user computing device 104 and / or be part of the user computing device. The user computing device 104 may include a mobile computing device (e.g., a smartphone or tablet computer), a desktop computer, a laptop computer, a smart wearable device, and / or a smart appliance. Additionally and / or alternatively, the user computing system may obtain data from one or more of the one or more user computing devices 104 and / or generate data using the one or more user computing devices. For example, a smartphone camera may be used to capture image data describing the environment, and / or an overlay application of the user computing device 104 may be used to track and / or process data provided to the user. Similarly, one or more sensors associated with a smart wearable device may be used to obtain data about the user and / or about the user's environment (e.g., a camera housed in the user's smart glasses may be used to obtain image data). Additionally and / or alternatively, data may be obtained and uploaded from other user devices that may be specifically used for data acquisition or generation.

[0168] The server computing system 130 includes one or more processors 132 and memory 134. The one or more processors 132 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be one processor or a plurality of processors operatively connected. The memory 134 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 134 may store data 136 and instructions 138 executed by the processor 132 to cause the server computing system 130 to perform operations.

[0169] In some implementations, the server computing system 130 includes or is otherwise implemented by one or more server computing devices. In instances where the server computing system 130 includes multiple server computing devices, such server computing devices may operate according to a sequential computing architecture, a parallel computing architecture, or some combination thereof.

[0170] As described above, the server computing system 130 may store or otherwise include one or more machine learning models 140. For example, the model 140 may be or may otherwise include various machine learning models. Example machine learning models include neural networks or other multi-layer nonlinear models. Example neural networks include feedforward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks. Fig. 15B Example model 140 is discussed.

[0171] Additionally and / or alternatively, server computing system 130 may include and / or be communicatively connected to a search engine 142 that may be used to crawl one or more databases (and / or resources). Search engine 142 may process data from user computing system 102, server computing system 130, and / or third-party computing system 150 to determine one or more search results associated with the input data. Search engine 142 may perform term-based search, tag-based search, Boolean-based search, image search, embedding-based search (e.g., nearest neighbor search), multimodal search, and / or one or more other search techniques.

[0172] The server computing system 130 may store and / or provide one or more user interfaces 144 for obtaining input data and / or providing output data to one or more users. The one or more user interfaces 144 may include one or more user interface elements, which may include input fields, navigation tools, content snippets, selectable tiles, widgets, data display carousels, dynamic animations, information pop-ups, image enhancements, text-to-speech, speech-to-text, augmented reality, virtual reality, feedback loops, and / or other interface elements.

[0173] User computing system 102 and / or server computing system 130 may train models 120 and / or 140 via interaction with a third-party computing system 150 communicatively coupled via network 180. Third-party computing system 150 may be separate from server computing system 130 or may be a part of server computing system 130. Alternatively and / or additionally, third-party computing system 150 may be associated with one or more web resources, one or more web platforms, one or more other users, and / or one or more contexts.

[0174] The third-party computing system 150 may include one or more processors 152 and a memory 154. The one or more processors 152 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be one processor or a plurality of processors operatively connected. The memory 154 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc., and combinations thereof. The memory 154 may store data 156 and instructions 158 that are executed by the processor 152 to cause the third-party computing system 150 to perform operations. In some implementations, the third-party computing system 150 includes or is otherwise implemented by one or more server computing devices.

[0175] The network 180 may be any type of communication network, such as a local area network (e.g., an intranet), a wide area network (e.g., the Internet), or some combination thereof, and may include any number of wired or wireless links. In general, communications over the network 180 may be conducted over any type of wired and / or wireless connection using a variety of communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and / or protection schemes (e.g., VPN, secure HTTP, SSL).

[0176] The machine learning models described in this specification can be used for a variety of tasks, applications, and / or use cases.

[0177] In some implementations, the input to the machine learning model of the present disclosure may be image data. The machine learning model may process the image data to generate an output. As an example, the machine learning model may process the image data to generate an image recognition output (e.g., recognition of the image data, potential embedding of the image data, encoded representation of the image data, hashing of the image data, etc.). As another example, the machine learning model may process the image data to generate an image segmentation output. As another example, the machine learning model may process the image data to generate an image classification output. As another example, the machine learning model may process the image data to generate an image data modification output (e.g., a change to the image data, etc.). As another example, the machine learning model may process the image data to generate an encoded image data output (e.g., an encoded and / or compressed representation of the image data, etc.). As another example, the machine learning model may process the image data to generate an amplified image data output. As another example, the machine learning model may process the image data to generate a prediction output.

[0178] In some implementations, the input to the machine learning model of the present disclosure may be text or natural language data. The machine learning model may process the text or natural language data to generate an output. As an example, the machine learning model may process the natural language data to generate a language encoding output. As another example, the machine learning model may process the text or natural language data to generate a potential text embedding output. As another example, the machine learning model may process the text or natural language data to generate a translation output. As another example, the machine learning model may process the text or natural language data to generate a classification output. As another example, the machine learning model may process the text or natural language data to generate a text segmentation output. As another example, the machine learning model may process the text or natural language data to generate a semantic intent output. As another example, the machine learning model may process the text or natural language data to generate an amplified text or natural language output (e.g., text or natural language data of higher quality than the input text or natural language, etc.). As another example, the machine learning model may process the text or natural language data to generate a prediction output.

[0179] In some implementations, the input to the machine learning model of the present disclosure may be speech data. The machine learning model may process the speech data to generate an output. As an example, the machine learning model may process the speech data to generate a speech recognition output. As another example, the machine learning model may process the speech data to generate a speech translation output. As another example, the machine learning model may process the speech data to generate a potential embedding output. As another example, the machine learning model may process the speech data to generate an encoded speech output (e.g., an encoded and / or compressed representation of the speech data, etc.). As another example, the machine learning model may process the speech data to generate an amplified speech output (e.g., speech data of higher quality than the input speech data, etc.). As another example, the machine learning model may process the speech data to generate a text representation output (e.g., a text representation of the input speech data, etc.). As another example, the machine learning model may process the speech data to generate a predicted output.

[0180] In some implementations, the input to the machine learning model of the present disclosure may be sensor data. The machine learning model may process the sensor data to generate an output. As an example, the machine learning model may process the sensor data to generate a recognition output. As another example, the machine learning model may process the sensor data to generate a prediction output. As another example, the machine learning model may process the sensor data to generate a classification output. As another example, the machine learning model may process the sensor data to generate a segmentation output. As another example, the machine learning model may process the sensor data to generate a segmentation output. As another example, the machine learning model may process the sensor data to generate a visualization output. As another example, the machine learning model may process the sensor data to generate a diagnostic output. As another example, the machine learning model may process the sensor data to generate a detection output.

[0181] In some cases, the input includes visual data, and the task is a computer vision task. In some cases, the input includes pixel data for one or more images, and the task is an image processing task. For example, the image processing task may be image classification, where the output is a set of scores, each score corresponding to a different object class and representing the possibility that one or more images depict an object belonging to the object class. The image processing task may be object detection, where the image processing output identifies one or more regions in one or more images, and for each region, identifies the possibility that the region depicts an object of interest. As another example, the image processing task may be image segmentation, where the image processing output defines, for each pixel in one or more images, a corresponding possibility for each category in a predetermined set of categories. For example, the set of categories may be foreground and background. As another example, the set of categories may be an object class. As another example, the image processing task may be depth estimation, where the image processing output defines a corresponding depth value for each pixel in one or more images. As another example, the image processing task may be motion estimation, where the network input includes multiple images, and the image processing output defines, for each pixel of one of the input images, the motion of the scene depicted at the pixel between images in the network input.

[0182] The user computing system may include multiple applications (e.g., applications 1 to N). Each application may include its own corresponding machine learning library and machine learning model. For example, each application may include a machine learning model. Example applications include text messaging applications, email applications, dictation applications, virtual keyboard applications, browser applications, etc.

[0183] Each application can communicate with multiple other components of the computing device (such as, for example, one or more sensors, a context manager, a device state component, and / or additional components). In some implementations, each application can communicate with each device component using an API (e.g., a public API). In some implementations, the API used by each application is specific to the application.

[0184] The user computing system 102 may include multiple applications (e.g., applications 1 to N). Each application communicates with a central intelligence layer. Example applications include text messaging applications, email applications, dictation applications, virtual keyboard applications, browser applications, etc. In some implementations, each application may communicate with the central intelligence layer (and the models stored therein) using an API (e.g., a public API across all applications).

[0185] The central intelligence layer can include multiple machine learning models. For example, a corresponding machine learning model (e.g., model) can be provided for each application and managed by the central intelligence layer. In other implementations, two or more applications can share a single machine learning model. For example, in some implementations, the central intelligence layer can provide a single model (e.g., a single model) for all applications. In some implementations, the central intelligence layer is included in or otherwise implemented by an operating system of the computing system 100.

[0186] The central intelligence layer can communicate with a central device data layer. The central device data layer can be a centralized data repository for the computing system 100. The central device data layer can communicate with many other components of the computing device (such as, for example, one or more sensors, context managers, device state components, and / or additional components). In some implementations, the central device data layer can communicate with each device component using an API (e.g., a private API).

[0187] Fig. 15BA block diagram of an example computing system 50 for performing link annotation prompts according to an example embodiment of the present disclosure is depicted. Specifically, the example computing system 50 may include one or more computing devices 52, which may be used to obtain and / or generate one or more data sets, which may be processed by a sensor processing system 60 and / or an output determination system 80 to be fed back to a user, who may provide information about features in one or more obtained data sets. The one or more data sets may include image data, text data, audio data, multimodal data, potentially encoded data, etc. The one or more data sets may be obtained via one or more sensors associated with the one or more computing devices 52 (e.g., one or more sensors in the computing devices 52). Additionally and / or alternatively, the one or more data sets may be stored data and / or retrieved data (e.g., data retrieved from a web resource). For example, a user may interact with images, text, and / or other content items. One or more determinations may then be generated using the interaction with the content items.

[0188] One or more computing devices 52 may obtain and / or generate one or more data sets based on image capture, sensor tracking, data storage retrieval, content download (e.g., downloading images or other content items from web resources via the Internet), and / or via one or more other technologies. One or more data sets may be processed using a sensor processing system 60. The sensor processing system 60 may perform one or more processing techniques using one or more machine learning models, one or more search engines, and / or one or more other processing techniques. One or more processing techniques may be performed in any combination and / or individually. One or more processing techniques may be performed serially and / or in parallel. Specifically, one or more data sets may be processed using a context determination block 62, which may determine a context associated with one or more content items. The context determination block 62 may identify and / or process metadata, user profile data (e.g., preferences, user search history, user browsing history, user purchase history, and / or user input data), previous interaction data, global trend data, location data, time data, and / or other data to determine a specific context associated with a user. A context may be associated with an event, a determined trend, a particular action, a particular type of data, a particular environment, and / or another context associated with a user and / or retrieved or obtained data.

[0189] The sensor processing system 60 may include an image pre-processing block 64. The image pre-processing block 64 may be used to adjust one or more values ​​of an acquired and / or received image to prepare the image to be processed by one or more machine learning models and / or one or more search engines 74. The image pre-processing block 64 may resize the image, adjust saturation values, adjust resolution, strip and / or add metadata, and / or perform one or more other operations.

[0190] In some implementations, the sensor processing system 60 may include one or more machine learning models, which may include a detection model 66, a segmentation model 68, a classification model 70, an embedding model 72, and / or one or more other machine learning models. For example, the sensor processing system 60 may include one or more detection models 66 that may be used to detect specific features in a processed dataset. Specifically, one or more detection models 66 may be used to process one or more images to generate one or more bounding boxes associated with detected features in the one or more images.

[0191] Additionally and / or alternatively, one or more segmentation models 68 may be utilized to segment one or more portions of a data set from one or more data sets. For example, one or more segmentation models 68 may utilize one or more segmentation masks (e.g., one or more segmentation masks generated manually and / or based on one or more bounding boxes) to segment a portion of an image, a portion of an audio file, and / or a portion of text. Segmentation may include isolating one or more detected objects and / or removing one or more detected objects from an image.

[0192] One or more classification models 70 may be used to process image data, text data, audio data, potentially encoded data, multimodal data, and / or other data to generate one or more classifications. One or more classification models 70 may include one or more image classification models, one or more object classification models, one or more text classification models, one or more audio classification models, and / or one or more other classification models. One or more classification models 70 may process data to determine one or more classifications.

[0193] In some implementations, one or more embedding models 72 may be used to process data to generate one or more embeddings. For example, one or more embedding models 72 may be used to process one or more images to generate one or more image embeddings in an embedding space. The one or more image embeddings may be associated with one or more image features of the one or more images. In some implementations, the one or more embedding models 72 may be configured to process multimodal data to generate multimodal embeddings. The one or more embeddings may be used to classify, search, and / or learn an embedding space distribution.

[0194] The sensor processing system 60 may include one or more search engines 74 that can be used to perform one or more searches. The one or more search engines 74 can crawl one or more databases (e.g., one or more local databases, one or more global databases, one or more private databases, one or more public databases, one or more dedicated databases, and / or one or more general databases) to determine one or more search results. The one or more search engines 74 can perform feature matching, text-based searches, embedding-based searches (e.g., k-nearest neighbor searches), metadata-based searches, multimodal searches, web resource searches, image searches, text searches, and / or application searches.

[0195] Additionally and / or alternatively, the sensor processing system 60 may include one or more multimodal processing blocks 76 that may be used to assist in processing multimodal data. The one or more multimodal processing blocks 76 may include generating multimodal queries and / or multimodal embeddings for processing by one or more machine learning models and / or one or more search engines 74.

[0196] The output of the sensor processing system 60 may then be processed using an output determination system 80 to determine one or more outputs to be provided to the user. The output determination system 80 may include a heuristic-based determination, a machine learning model-based determination, a user selection-based determination, and / or a context-based determination.

[0197] The output determination system 80 can determine how and / or where to provide one or more search results in the search result interface 82. Additionally and / or alternatively, the output determination system 80 can determine how and / or where to provide one or more machine learning model outputs in the machine learning model output interface 84. In some implementations, one or more search results and / or one or more machine learning model outputs can be provided for display via one or more user interface elements. One or more user interface elements can be superimposed on the displayed data. For example, one or more detection indicators can be superimposed on the detected object in the viewfinder. One or more user interface elements can be selected to perform one or more additional searches and / or one or more additional machine learning model processes. In some implementations, user interface elements can be provided as dedicated user interface elements for specific applications and / or can be uniformly provided across different applications. One or more user interface elements may include pop-up displays, interface overlays, interface tiles and / or small pieces, carousel interfaces, audio feedback, animations, interactive widgets, and / or other user interface elements.

[0198] Additionally and / or alternatively, data associated with the output of the sensor processing system 60 may be utilized to generate and / or provide an augmented reality experience and / or a virtual reality experience 86. For example, one or more acquired data sets may be processed to generate one or more augmented reality rendering assets and / or one or more virtual reality rendering assets, which may then be used to provide an augmented reality experience and / or a virtual reality experience 86 to a user. The augmented reality experience may render information associated with an environment into the corresponding environment. Alternatively and / or additionally, objects associated with the processed data sets may be rendered into the user environment and / or the virtual environment. Rendering data set generation may include training one or more neural radiance field models to learn a three-dimensional representation of one or more objects.

[0199] In some implementations, one or more action prompts 88 may be determined based on the output of the sensor processing system 60. For example, a search prompt, a purchase prompt, a generate prompt, a reservation prompt, a call prompt, a redirect prompt, and / or one or more other prompts may be determined to be associated with the output of the sensor processing system 60. The one or more action prompts 88 may then be provided to the user via one or more selectable user interface elements. In response to selection of the one or more selectable user interface elements, a corresponding action of the corresponding action prompt may be performed (e.g., a search may be performed, a purchase application programming interface may be utilized, and / or another application may be opened).

[0200] In some implementations, one or more generative models 90 may be used to process one or more data sets and / or output of the sensor processing system 60 to generate model-generated content items that may then be provided to a user. The generation may be prompted based on a user selection and / or may be performed automatically (e.g., automatically based on one or more conditions, which may be associated with a threshold amount of search results that are not represented).

[0201] One or more generative models 90 may include a language model (e.g., a large language model and / or a visual language model), an image generation model (e.g., a text-to-image generation model and / or an image enhancement model), an audio generation model, a video generation model, a graphics generation model, and / or other data generation models (e.g., other content generation models). One or more generative models 90 may include one or more Transformer models, one or more convolutional neural networks, one or more recurrent neural networks, one or more feedforward neural networks, one or more generative adversarial networks, one or more self-attention models, one or more embedding models, one or more encoders, one or more decoders, and / or one or more other models. In some implementations, one or more generative models 90 may include one or more autoregressive models (e.g., a machine learning model trained to generate predicted values ​​based on prior behavior data) and / or one or more diffusion models (e.g., a machine learning model trained to generate predicted data based on generating and processing distribution data associated with input data).

[0202] One or more generative models 90 can be trained to process input data and generate model-generated content items, which can include multiple predicted words, pixels, signals and / or other data. Model-generated content items can include novel content items that are different from any existing works. One or more generative models 90 can utilize learned representations, sequences and / or probability distributions to generate content items, which can include phrases, storylines, settings, objects, characters, beats, lyrics, and / or other aspects not included in existing content items.

[0203] The one or more generative models 90 may include a visual-linguistic model.

[0204] The visual language model can be trained, tuned, and / or configured to process image data and / or text data to generate natural language output. The visual language model can utilize a pre-trained large language model (e.g., a large autoregressive language model) and one or more encoders (e.g., one or more image encoders and / or one or more text encoders) to provide detailed natural language output that simulates natural language written by humans.

[0205] The visual language model can be used for zero-shot image classification, few-shot image classification, image captioning, multimodal query refinement, multimodal question answering, and / or can be tuned and / or trained for multiple different tasks. The visual language model can perform visual question answering, image caption generation, feature detection (e.g., content monitoring (e.g., for inappropriate content)), object detection, scene recognition, and / or other tasks.

[0206] The visual language model can utilize a pre-trained language model and then tune it to achieve multimodality. The training and / or tuning of the visual language model can include image-text matching, masked language modeling, multimodal fusion of cross attention, contrastive learning, prefix language model training, and / or other training techniques. For example, the visual language model can be trained to process images to generate predicted text similar to real-value text data (e.g., real-value subtitles of images). In some implementations, the visual language model can be trained to replace the masked word units of the natural language template with text word units describing the features depicted in the input image. Alternatively and / or additionally, training, tuning, and / or model reasoning can include multi-layer connections of visual and text embedding features. In some implementations, the visual language model can be trained and / or tuned by jointly learning image embedding and text embedding generation, which can include training and / or tuning the system to map the embedding to a joint feature embedding space that maps text features and image features to a shared embedding space. Joint training can include parallel embedding of image-text pairs and / or can include triple training. In some implementations, images can be utilized and / or processed as a prefix to a language model.

[0207] The output determination system 80 may process one or more data sets and / or the output of the sensor processing system 60 to generate enhanced data using a data enhancement block 92. For example, one or more images may be processed using the data enhancement block 92 to generate one or more enhanced images. Data enhancement may include data correction, data cropping, removal of one or more features, addition of one or more features, resolution adjustment, lighting adjustment, saturation adjustment, and / or other enhancements.

[0208] In some implementations, one or more data sets and / or outputs of the sensor processing system 60 may be stored based on the determination of the data storage block 94 .

[0209] The output of the output determination system 80 may then be provided to the user via one or more output components of the user computing device 52. For example, one or more user interface elements associated with the one or more outputs may be provided for display via a visual display of the user computing device 52.

[0210] The process may be performed iteratively and / or continuously. One or more user inputs to the provided user interface elements may modulate and / or influence the continuous processing loop.

[0211] The technology discussed herein relates to servers, databases, software applications, and other computer-based systems and the actions taken and the information sent to and from these systems. The inherent flexibility of computer-based systems allows for multiple possible configurations, combinations, and partitioning of tasks and functions between and among components. For example, the processes discussed herein can be implemented using a single device or component or multiple devices or components working in combination. Databases and applications can be implemented on a single system or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0212] Although the subject matter has been described in detail with respect to various specific example embodiments of the subject matter, each example is provided by way of explanation, rather than limitation of the present disclosure. Those skilled in the art, after understanding the foregoing, may easily produce changes, modifications, and equivalents to these embodiments. Therefore, the present disclosure does not exclude such modifications, changes, and / or additions to the subject matter that would be readily understood by those of ordinary skill in the art. For example, a feature shown or described as part of one embodiment may be used together with another embodiment to produce yet another embodiment. Therefore, the present disclosure is intended to cover these changes, modifications, and equivalents.

Claims

1. A computing system for review prompt generation and input retrieval, the system comprising: one or more processors; as well as one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations comprising: obtaining content data, wherein the content data is associated with a web resource; processing the content data with a generative model to generate a predictive hint, wherein the hint includes a predicted text string associated with reviewing the web resource; providing the prediction prompt for display using an input prompt interface, wherein the input prompt interface is configured to receive input; obtaining comment input data from a user computing system via the input prompt interface, wherein the comment input data includes a user-generated comment on the web resource; and Data associated with the review input data is stored with data associated with the web resource, wherein the data associated with the review input data is stored in a searchable database to be provided for display in response to the web resource being provided as a search result.

2. The system of claim 1, wherein the operations further comprise: obtaining user data, wherein the user data is associated with a particular user, wherein the user computing system is associated with the particular user; and Wherein processing the content data with the generative model to generate the predictive prompt comprises: processing the content data and the user data with the generative model.

3. The system of claim 2, wherein the user data includes user search history data, and wherein the generative model generates the predictive hint based on the particular user's previous searches for information associated with the subject matter of the web resource.

4. The system of claim 2, wherein the user data comprises user browser history data, and wherein the generative model generates the predictive hint based on the particular user's previous viewing of other web resources that include information associated with the subject matter of the web resource.

5. The system of claim 2, wherein the operations further comprise: generating a graphics card based on the user data, the content data, and the comment input data, wherein the graphics card includes a user profile identifier for the particular user and data associated with the comment input data; and The graphics card is stored.

6. The system of claim 5, wherein the graphics card includes a graphics background generated based on the comment input data using an image generation model.

7. The system of claim 1, wherein the operations further comprise: Get the search query; determining that the web resource is associated with the search query; A specific search result is provided for display, wherein the specific search result includes a link to the web resource, a title of the web resource, and data associated with the review input data.

8. The system of claim 1, wherein storing the data associated with the review input data together with the data associated with the web resource comprises: Generate web resource annotations; as well as The web resource annotation is stored with a plurality of other web resource annotations associated with the web resource.

9. The system of claim 8, wherein the operations further comprise: The web resource annotation and the plurality of other web resource annotations are provided in an annotation interface, the annotation interface providing the web resource annotation and the plurality of other web resource annotations in a plurality of graphic cards.

10. The system of claim 1, wherein the generative model comprises an autoregressive language model, and wherein the generative model is prompted to generate questions describing requests for information about the web resource.

11. A computer-implemented method for linking annotation prompts, the method comprising: obtaining, by a computing system including one or more processors, contextual data, wherein the contextual data is associated with a particular content display instance, wherein the particular content display instance includes a particular user viewing a particular content item; determining, by the computing system, an input request action based on the context data, wherein the input request action comprises providing an input entry interface to a user to obtain a user input; processing, by the computing system, the context data with a generative language model to generate a predictive prompt, wherein the predictive prompt comprises a natural language request for information generated based on the context data; The computing system provides the prediction prompt in the input entry interface; Obtaining user-generated content by the computing system via the input entry interface; as well as A link annotation is generated by the computing system based on the user-generated content, wherein the link annotation is generated to be provided for display in a search results interface in response to the particular content item being determined as a search result.

12. The method of claim 11, wherein the context data is associated with a type of content being provided for display.

13. The method of claim 11, wherein the contextual data is associated with the particular user associated with the particular content display instance, wherein the contextual data includes search history data.

14. The method of claim 11, wherein the content being provided for display is associated with a specific web resource, wherein the context data is associated with interaction data for links to the specific web resource on multiple social networking platforms, and wherein the input request action is determined based on the interaction data.

15. The method of claim 11, wherein the context data includes user data and content data, and wherein the input request operation is based on a topic associated with the content being provided for display is determined based on one of a plurality of topics that the particular user is known to determine based on the user data.

16. The method of claim 11, wherein the context data includes previous annotations generated by the particular user, and wherein the predictive hint includes a structure based on a previous structure of the previous annotations.

17. One or more non-transitory computer-readable media collectively storing instructions that, when executed by one or more computing devices, cause the one or more computing devices to perform operations comprising: Get the first search query at the first time; determining web resources responsive to the first search query; obtaining content data, wherein the content data is associated with the web resource; processing the content data with a generative model to generate a predictive hint, wherein the hint includes a predicted text string associated with reviewing the web resource; Providing the prediction prompt for display in an input prompt interface, wherein the input prompt interface includes an input entry box; obtaining comment input data from a user computing system via the input prompt interface, wherein the comment input data includes user-generated content; storing the user-generated content; obtaining a second search query at a second time, wherein the second time is different from the first time; determining that the web resource is responsive to the second search query; as well as The user-generated content is provided in a search results interface with data describing the web resource.

18. The one or more non-transitory computer-readable media of claim 17, wherein the first search query and the second search query are different.

19. The one or more non-transitory computer-readable media of claim 17, wherein the review input data comprises multimodal data.

20. The one or more non-transitory computer-readable media of claim 19, wherein the multimodal data includes text data and image data.

Images