Query intent understanding and search result generation

Abstract

Systems and methods for performing a search based on chat data may include determining search results for a user query by utilizing a generative language model to generate multiple rounds of perceptual queries to include additional details associated with a determination intent of the query. The generative language model may receive a user query indicating a chat session of the user and multiple rounds of chat data, and determine an intent of the user query based on the user query and the multiple rounds of chat data. The generative language model may generate multiple rounds of perceptual queries to include details associated with the determination intent of the user query by rewriting the user query, and the multiple rounds of perceptual queries may be used for search result determination. A generative language model may be utilized to tune an embedded model that may be used for query intent determination.

Description

Technical Field

[0001] The present disclosure generally relates to query intent determination. More specifically, the present disclosure relates to utilizing a generative model to determine the intent of a query based on a chat session history, which intent can then be utilized to generate a context-aware query. Background Art

[0002] With the development of artificial intelligence (AI) chatbots, chat interfaces and chat communications can be common tools for AI chatbot interactions and in some cases for Internet searches. When a user interacts with a system via a chat interface and generates a history of inputs and responses, each message may become dependent on previous inputs and / or responses, and therefore may not be easily understood individually. However, a chat may end with executable questions and / or commands. Questions and / or commands may not be understood when processed separately from other messages.

[0003] Traditional search queries may be crafted to include descriptive information to generate accurate responses. Using a chat interface, users may begin to omit information provided in previous responses in their queries, which may include using pronouns and / or discussing details without indicating a general topic. Without the context of a history of input and responses, it may be difficult for the system to accurately determine the user's intent for a given query. When the search engine cannot accurately determine the intent of the user's query, the search engine may identify and / or provide search results that may not be responsive to the user's intent. The system may not be able to generate content items and responses that reflect what the user is requesting. Summary of the invention

[0004] Aspects and advantages of embodiments of the present disclosure will be set forth in part in the following description, or may be learned from the description, or may be learned through practice of the embodiments.

[0005] An example aspect of the present disclosure relates to a computer-implemented method for generating a context-aware query. The method may include obtaining an input query by a computing system including one or more computing devices. The method may include obtaining multi-round query data by the computing system, which may describe previous inputs obtained before the input query. The previous input and the input query may be associated with a specific multi-round session. The method may include processing the input query and the multi-round query data by the computing system to generate a context-aware query. The context-aware query may describe the input query and additional details. The additional details may describe the context of the input query associated with the multi-round query data. The method may include processing the context-aware query by the computing system using a machine learning embedding model to generate a query embedding. The method may include determining, by the computing system, a query embedding cluster associated with the query embedding. The query embedding cluster may be associated with multiple other embeddings associated with multiple other queries. The method may include determining, by the computing system, multiple search results based on the query cluster.

[0006] In some implementations, the method may include determining, by a computing system and based on the query embedding cluster, one or more attributes associated with the query embedding. The one or more attributes may describe a specific topic associated with at least one of the input query or the multiple query data.

[0007] In some implementations, determining a query embedding cluster associated with the query embedding may include mapping, by a computing system, the query intent pairs to an embedding space. Determining a query embedding cluster associated with the query embedding may include determining, by the computing system and based on at least one of the query embedding or the intent embedding, that the query intent pairs are associated with a plurality of other embeddings associated with nodes within the query graph.

[0008] In some implementations, the method may include determining, by the computing system, a plurality of media content items based on the query embedding cluster.

[0009] Another example aspect of the present disclosure relates to a computing system for generating multi-round perception queries. The system may include one or more processors and one or more non-transitory computer-readable media that store instructions together, which when executed by one or more processors cause the computing system to perform operations. The operation may include obtaining an input query. The operation may include obtaining multi-round query data. The multi-round query data may describe a previous input obtained before the input query. The previous input and the input query may be associated with a specific multi-round session. The operation may include processing the input query and the multi-round query data using a machine learning language model to generate a multi-round perception query. The multi-round perception query may describe an input query enhanced with additional details determined based on the multi-round query data. The operation may include processing the multi-round perception query using a machine learning embedding model to generate a query embedding. The operation may include determining a query embedding cluster associated with the query embedding. The operation may include determining multiple search results based on the query cluster.

[0010] Another example aspect of the present disclosure relates to one or more non-transitory computer-readable media that collectively store instructions that, when executed by a computing system, cause the computing system to perform operations. The operations may include obtaining input data, the input data including a query. The operations may include processing the query using an embedding model to generate a query embedding. The operations may include processing the input data using a generative model to determine a query intent, which query intent may describe the type of information requested. The operations may include obtaining one or more second query embeddings associated with one or more second queries having one or more second query intents based on the query intent. The one or more second query intents may be associated with the query intent of the input data. The operations may include evaluating a loss function that evaluates the difference between the query embedding and the one or more second query embeddings. The operations may include adjusting one or more parameters of the embedding model based at least in part on the loss function.

[0011] In some implementations, the query may include a rewritten query generated by obtaining an input query and multi-turn query data. The multi-turn query data may describe previous input obtained before the input query. The previous input and the input query may be associated with a specific multi-turn session. The rewritten query may be generated by processing the input query and the multi-turn query data using a language model to generate the rewritten query.

[0012] In some implementations, operations may include determining that an intent embedding is associated with a query intent, and one or more second query embeddings may be obtained based on the intent embedding.

[0013] In some implementations, operations may include generating a reconstructed data graph of query clusters based on the query embeddings, the intent embeddings, and the one or more query embeddings.

[0014] Other aspects of the disclosure relate to various systems, apparatuses, non-transitory computer-readable media, user interfaces, and electronic devices.

[0015] These and other features, aspects and advantages of various embodiments of the present disclosure will be better understood with reference to the following description and appended claims.The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate example embodiments of the present disclosure and, together with the description, serve to explain the relevant principles. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] A detailed discussion of embodiments for those of ordinary skill in the art is set forth in this specification with reference to the accompanying drawings, in which:

[0017] Figure 1 Depicted is a generalized block diagram of an example query intent determination system according to an example embodiment of the present disclosure.

[0018] Figure 2 Depicted is a detailed block diagram of an example query intent determination system according to an example embodiment of the present disclosure.

[0019] Figure 3 Depicted is a flow diagram of an example method for performing context-aware query generation and search result determination according to an example embodiment of the present disclosure.

[0020] Figure 4 Depicted is a block diagram of an example query rewrite system that generates context-aware queries according to example aspects of the present disclosure.

[0021] Figure 5 Depicted is a block diagram of example components of a search result determination system according to an example embodiment of the present disclosure.

[0022] Figure 6 Depicted is a block diagram of an example embedding model training system according to an example embodiment of the present disclosure.

[0023] Figure 7 Depicted is a block diagram of an example intent attribute determination system according to an example embodiment of the present disclosure.

[0024] Figure 8 Depicted is a flow chart of an example method for determining search results according to an example embodiment of the present disclosure.

[0025] Fig. 9 A flow chart of an example method performed according to an example embodiment of the present disclosure is depicted.

[0026] Fig. 10A A block diagram of an example computing system that performs search result generation according to an example embodiment of the present disclosure is depicted.

[0027] Fig. 10BA block diagram of an example computing system that performs search result generation according to an example embodiment of the present disclosure is depicted.

[0028] Reference numerals repeated among the various figures are intended to identify like features in the various implementations. DETAILED DESCRIPTION

[0029] In general, the present disclosure relates to systems and methods for generating search results for queries based on context-aware queries generated using generative models and based on multiple rounds of chat sessions. Specifically, the systems and methods disclosed herein may utilize a generative language model (e.g., a large language model (LLM)) to process an input query and data associated with the input query to generate a context-aware query including detailed information associated with the query and the query context. For example, the generative language model may process previous messages and responses in a user's chat session together with the query and determine the intent of the query. The generative language model may generate multiple rounds of perceptual queries including the intent of the determined query. For example, a user may interact with a chatbot via a chat-like interface and may exchange multiple messages. As the user continues to interact with the system, their messages may depend on topics (and / or details) and / or responses within previous messages, which may result in uncertainty about the intent of the message when viewing the message alone.

[0030] In order to determine the intent of the message, the system may be able to utilize a generative language model to process the user's current message and chat session history. The generative language model may determine the semantics of the message and chat session history and generate a multi-turn aware message (e.g., a context-aware query). The multi-turn aware message may describe the intent of the initial message and the chat session history. The generative model may utilize the language understanding, semantic analysis, and generation capabilities of the generative language model to generate the multi-turn aware message. Using the multi-turn aware message, the system may generate a response and / or media content item based on the multi-turn aware message, and may provide the determined and / or generated content item as a response to the user's input message.

[0031] In order to generate responses and / or media content items for multi-round perception queries, the system may utilize a data graph (e.g., a query embedding cluster graph, a query graph, an intent graph, and / or a query intent graph). The data graph (and / or intent graph) may include a graph representation associated with learned query clusters and learned intent clusters. The query embedding cluster graph may include clusters of similar queries linked together based on common attributes such as intent. The system and method may utilize an embedding model to process multi-round perception queries to determine query embeddings within the intent graph of the multi-round perception query. The system and method may then process the query embedding to determine the intent of the query based at least in part on determining an association with one or more query embedding clusters. Based on the associated embedding clusters, the system and method may determine, generate, and / or obtain search results and / or media content items.

[0032] Some aspects of the present disclosure may involve training and / or tuning a machine learning model based on an intent determination from a provided query. Specifically, the intent of a provided query may be determined via generating a language model (e.g., an LLM, a visual language model (VLM), etc.), and the determined intent may be used to evaluate a loss function for training a query embedding model and / or adjusting an intent graph (and / or a task graph that maps query clusters to specific tasks). The query embedding model may process a query as input and generate a query embedding that maps the query to an embedding space associated with an intent graph, the intent graph including a plurality of learned distributions and / or query clusters associated with the intent graph. For example, a query embedding model may process a query and map the query to an intent embedding space associated with a query associated with a similar query intent.

[0033] In some implementations, an embedding model may be tuned using a generative language model and a loss function. The loss function may process a query embedding and an intent determination from a generative model. The loss function may determine a loss between a query embedding and an intent determination, which may be used to improve the query embedding model. For example, a gradient descent of the loss between a query embedding and an intent determination may be back-propagated to a query embedding model to adjust one or more parameters of the query embedding model. The embedding model may be trained and / or tuned to generate query embeddings that are associated with (e.g., close to and / or similar to) an embedding of an intent associated with a query, as well as other query embeddings with similar intents. By leveraging the intent determination of a generative model, an embedding model may be trained to generate similar embeddings for queries and corresponding intents for queries, which may encourage intent-based distributions.

[0034] Chat interfaces can provide a more immersive experience for user interactions that obtain information via a network platform. However, chat interfaces may have difficulty understanding large-scale intent when processing queries and / or prompts. Chat interfaces can encourage users to interact with their systems in a more conversational manner than traditional search systems. Chat interfaces can enable users to provide multiple messages (and / or queries), each of which can be related to each other based on similar topics and / or based on the construction of previous messages and / or topics. As users continue to participate in a query session, multiple messages (e.g., user messages and chatbot responses) that can begin to reference previous messages can be generated and / or received, while in traditional search systems, each query provided by a user can be considered an independent session without being associated with a previous session. If a user relies on and / or references topics or details from the history of queries and responses, it may be difficult to discern the exact intent of their future queries when viewing the text of future queries alone. For example, a user may refer to an object in proper noun form at one point in a query session, but may later refer to the object using a pronoun because they have already mentioned the object in proper noun form within the session (e.g., "Please tell me about brand X's new smartphone," followed by "it?"). Without knowing the earlier query using the proper noun, it may be difficult for the search system to determine the intent of the later query using the pronoun. In a traditional one-shot search system, a user may only use the proper noun form of an object because there is no history of queries for the user to refer to.

[0035] Rewriting queries using a generative language model can eliminate the difficulty of determining the intent of a query from a chat-like interface. A generative language model can be used to interpret, understand, and textualize a given chat session. A generative language model can be trained on multiple natural language processing tasks, and can be trained to perform semantic understanding and conditional text string generation. In some implementations, the generative language model may include an autoregressive language model that is trained to learn sequence representations to predict the next word in a sequence based on previous words and / or previous data in an input data set. Therefore, the generative language model may be able to understand a wide range of language contexts and requests. For example, the generative model may be able to determine the connection between different excerpts such as text (e.g., the intent of a text) based on the content and context of another text. More specifically, the generative language model may be able to determine the intent of a given query based on the history of the query and its response. For example, a query from a chat-like interface may be provided to the generative model, and the chat-like interface includes a history of the query and its response. The generative model can process data and rewrite the query so that the rewritten query reflects the intent of the query based on the history of the query and its response. The generative model may determine a common topic between a query and a history of queries and responses, and include details associated with the common topic in the query to generate a rewritten query.

[0036] The generative language model can improve search result relevance and certainty by processing existing queries within the intent graph and determining related queries and associations within the intent graph. The intent graph can include multiple clusters, where each cluster can include multiple associated query embeddings and can be associated with multiple nodes that link similar clusters together. The generative language model can determine the nuances and intents of the query embeddings within the clusters and discover clusters that are similar to the query embeddings. Based on the discovered similar clusters, new boundaries can be generated between the query embedding's cluster and the discovered similar clusters.

[0037] Using intent determinations from a generative language model to train a query embedding model can improve model performance and output accuracy by reducing iterative search instances, thereby reducing the computational cost for processing received queries. For example, a query embedding model can be trained using a loss function that generates a loss between a query embedding generated for a given input from a query embedding model and an intent determination for a given input from a generative language model. The loss can be provided to the query embedding model to train or tune one or more parameters of the query embedding model. A query embedding model with improved accuracy can reduce iterations for determining and generating results to satisfy a received query. For example, a less accurate embedding model may result in a user having to provide multiple queries about the same question to obtain a desired answer, while an embedding model tuned based on a generative language model intent determination can obtain the desired answer on the first try, which can reduce search instances.

[0038] The system and method of the present disclosure provide many technical effects and benefits. As an example, the system and method can be used to determine search results in response to the intent of a multi-round chat session. Additionally and / or alternatively, the system and method can use a generative language model to train and / or tune a query embedding model to generate a query embedding that clusters the embedding based on the intent. The learned intent distribution of the multi-round perceived query and query embedding cluster can be used to determine, generate and / or obtain search results and / or media content items in response to the intent of the chat session without manually rewriting the query and / or without iterative search and refinement. The computing system may require fewer computing resources to satisfy their requests when provided with a more accurate response to a given query, and may abandon additional search queries that are usually necessary to satisfy their requests. Additionally or alternatively, using a more accurate query embedding model, the system can have improved computing efficiency in satisfying user requests. For example, a more accurate query embedding can reduce iterations within the system for satisfying user requests (e.g., the system iterates query embedding until an acceptable solution is generated).

[0039] The system and method of the present disclosure provide many technical effects and benefits. Specifically, the implementation of the present disclosure can reduce server-side resource utilization when performing user searches by generating and utilizing context-aware queries to retrieve search results for user queries. Using context-aware queries to determine responses can provide responses with higher accuracy and alleviate further iterations of queries and responses used to satisfy user searches.

[0040] For example, conventional search techniques may exhibit suboptimal performance when searching for queries related to nuanced topics, niche interests, technically complex topics, etc. Suboptimal performance, in turn, may require a user to perform multiple search iterations to refine a query based on initial results, resulting in a significant increase in network bandwidth utilization and computing resource utilization of the server performing the search. However, by utilizing contextual information and generating a language model, implementations of the present disclosure may significantly reduce, eliminate, and / or mitigate the need to perform multiple search iterations to refine a query, thereby avoiding bandwidth utilization and computing resource utilization associated with such iterations.

[0041] Referring now to the accompanying drawings, example embodiments of the present disclosure will be discussed in more detail.

[0042] Figure 1A generalized block diagram of an example query intent determination system 200 according to an example implementation of the present disclosure is depicted. In some implementations, the system 200 receives as input an input query 202 and multi-turn query data 204. In some cases, the input query 202 may indicate a user's search query. The multi-turn query data 204 may include multiple messages (e.g., user search queries and / or prompts to a chatbot (e.g., a chatbot supporting a large language model)) and responses thereto. In some cases, the multiple messages and responses thereto may have been made by the user and are associated with the input query 202. For example, the input query 202 may be a new user query after multiple user messages and chatbot responses within the multi-turn query data 204. The input query 202 and the multi-turn query data 204 may be processed using a generative language model 206 (e.g., a large language model) to determine the intent of the input query 202 based on the multi-turn query data 204. For example, the input query 202 may include a general query using pronouns that are not interpretable without additional context (e.g., without a chat session history). The multi-turn query data 204 may include user messages (e.g., previous user search queries and / or user prompts) and / or responses (e.g., chatbot responses) that include nouns corresponding to pronouns used in the input query 202. The generated language model 206 may generate as output a context-aware query 208 (e.g., a multi-turn aware query) that indicates the intent of the input query 202 relative to the multi-turn query data 204. The context-aware query 208 may be generated to include the semantics of the input query 202 while adding additional details determined based on the chat session history of the multi-turn query data 204. For example, the input query 202 may include a statement such as "how much does it cost?", and the multi-turn query data 204 may show a common theme of "new smartphones". Therefore, the generated language model 206 may generate a context-aware query 208 that includes "how much does a new smartphone cost?", thereby interpreting the intent of the input query 202. The context-aware query 208 may then be used instead of and / or in addition to the input query 202 to determine the search results 210. In some implementations, search results 210 can include media content items and / or related web search results.

[0043] Figure 2A detailed block diagram of an example query intent determination system 300 according to an example implementation of the present disclosure is depicted. The system 300 receives an input query 202 and multi-turn query data 204 as input, and generates search results 210. In some implementations, the multi-turn query data 204 may include multiple queries and responses 304. For example, the input query 202 may be a query from a chat-style search interface, and the multi-turn query data 204 may include multiple user messages (e.g., multiple user search queries and / or multiple user prompts entered into a chatbot interface) and responses 304 made in the chat-style interface before the input query 202 (e.g., chatbot responses generated using a language model). In some implementations, the input query 202 may include multimodal data, which includes two or more different types of data. For example, the input query 202 may include image data, text data, audio data, video data, potentially encoded data, statistical data, and / or multimodal data. In some implementations, the multi-turn query data 204 including multiple queries and responses 304 may include image data and text data that may be associated with each other. Generative models can be configured and / or trained to process multimodal data (e.g., vision-language models).

[0044] The input query 202 and the multi-turn query data 204 may be processed using a generative language model (e.g., a large language model) 206 to generate a context-aware query 208. The generative language model 206 may be a generative language model trained on tasks and / or topics of great breadth and depth (e.g., the generative model may be trained on multiple natural language processing tasks). For example, the generative language model may be trained on multiple tasks, rather than on any one specific operation or task (e.g., generating a natural language response). The generative language model 206 may be trained to generate API calls that may be processed serially and / or in parallel with the context-aware query 208. The context-aware query 208 may be a rewritten version of the input query 202 to include additional details associated with the context of the multi-turn query data 204 and the multiple messages and responses 304. For example, the input query 202 may include language that is dependent on previous user messages and / or chatbot responses within the multi-turn query data 204. For example, a pronoun within the input query 202 is intended to refer to a noun within the multi-turn query data 204. Generative language model 206 can rewrite input query 202 to incorporate expected references within multi-pass query data 204 , and context-aware query 208 can include all necessary information to accurately determine the intent of original input query 202 .

[0045] The context-aware query 208 may be processed using a query embedding model 306 (e.g., a machine learning embedding model) to generate a query embedding 310. The query embedding 310 may describe multiple values ​​associated with multiple features of the context-aware query 208. The query embedding 310 may map the query to an embedding space associated with the data graph. For example, the query embedding model 306 may generate a query embedding 310 that maps the context-aware query 208 to a query cluster in the embedding space. The query cluster may be associated with a learned distribution within the intent graph 308. In some implementations, the query embedding model 306 may be trained to generate query embeddings 310 that cluster queries with similar query intents. For example, the query embedding model 306 may be trained to generate query embeddings with similar values ​​for queries identified as having the same intent (e.g., "How much is a new brand Y controller?" and "Price of brand Y controller" may be processed to generate similar query embeddings based on shared query intent). The query embedding model 306 may generate multiple query embeddings, which may be clustered into multiple query embedding clusters 309. The plurality of query embedding clusters 309 may be processed to determine a plurality of learned distributions, which may be learned and utilized to generate an intent graph 308 .

[0046] The query embedding 310 may describe multiple features associated with the context-aware query 208. The query embedding 310 may be associated with one or more query embedding clusters 309, which may be associated with one or more intents in the intent graph 308. The intent graph 308 may include multiple nodes and boundaries, and may include a graphical representation that identifies similar queries with similar intents. The intent graph 308 may include a graph data structure containing multiple boundaries and nodes. In some implementations, a node of the intent graph 308 may include one or more embedding clusters 309 indicating a set of similar query embeddings. For example, the query embedding 310 may map the context-aware query 208 to a specific embedding cluster in the embedding cluster 309 that includes similar queries with similar intents. The boundaries between the embedding clusters 309 may link the clusters 309 to closely related intents and / or other attributes (e.g., topics, themes, etc.) based on the queries within the clusters 309. In some implementations, the boundaries may be used to determine subsequent query suggestions because nodes associated with similar topics may be linked via boundaries. For example, an embedded cluster associated with the price of a particular product may be associated with a node that is linked via an edge of the intent graph 308 to a node associated with a location for purchasing the particular product.

[0047] The query embedding cluster 309 can be used to determine and / or generate search results 210. For example, the embedding cluster associated with the query embedding 310 can include the query that has been understood and its intention. The understood query can be used to generate the search results of the context-aware query 208 and the input query 202 based on its determined similarity. The search results 210 may include media content items related to the embedding cluster associated with the query embedding 310. In one implementation, the search results 210 may include multimodal data, which includes two or more types of data. For example, the search results 210 may include text data and image data related to the embedding cluster to which the context-aware query 208 is mapped.

[0048] In some implementations, the query embedding cluster 309 may be utilized to determine search results 210 in response to the input query 202 by utilizing a search engine to process the context-aware query 208 and / or one or more other queries in the embedding cluster 309 to determine a plurality of search results. Alternatively and / or additionally, one or more content items may be tagged and / or indexed in association with a particular intent and / or a particular embedding cluster. The tagged and / or indexed content items may be provided as search results for queries determined to be associated with a particular query embedding cluster. For example, a reliable encyclopedia entry content item may be tagged so that the reliable encyclopedia entry content item may be utilized to process and respond to queries associated with a particular topic (e.g., Vancouver Island marmot). One or more content items may be tagged to different nodes of the intent graph 308 (and / or task graph) to provide a unified presentation of content items when a query embedding associated with a particular embedding cluster is received.

[0049] Figure 3 A flow chart of an example method 320 for performing context-aware query generation and search result determination according to an example implementation of the present disclosure is depicted. Figure 3 The steps performed in a specific order are depicted for the purpose of illustration and discussion, but the method of the present disclosure is not limited to the specific illustrated order or arrangement. The various steps of the method 320 may be omitted, rearranged, combined and / or adjusted in various ways without departing from the scope of the present disclosure.

[0050] At 322, the computing system may obtain an input query. The input query may include a message within a chat interface. The input query may be preceded by multiple other messages in the chat session. The multiple other messages may include user messages (e.g., user search queries and / or user prompts) and / or chatbot messages (e.g., responses generated by a generated language model). In some implementations, the input query may include multimodal data, which includes two or more different types of data. For example, the input query may include text data and image data (e.g., a query including "How many calories does this have compared to the other one?" and an image of a box of cereal).

[0051] At 324, the computing system may obtain multi-turn query data. In some implementations, the multi-turn query data may describe previous inputs obtained before the query was input, and the previous inputs may be associated with the same multi-turn session as the input query (e.g., a chat session history associated with a particular chat session instance). For example, a user may be engaged in a chatty search session, and the input query may be the user's most recent query. The multi-turn query data may include multiple messages (e.g., multiple queries and responses) exchanged (e.g., provided, received, and / or generated) in the search session prior to the user's most recent query. The multi-turn query data may describe a chat session history including multiple user messages and multiple chatbot responses (e.g., responses generated using a language model that processes the user's messages).

[0052] At 326, the computing system may process the input query and the multi-turn query data to generate a context-aware query. The context-aware query may include the semantics of the input query with additional details indicating the context of the input query relative to the multi-turn query data (e.g., the generative model may replace dependent pronouns within the input query with corresponding nouns determined based on the chat session history of processing the multi-turn query data). In some implementations, where the input query may be multimodal data, the context-aware query may be determined based on image data and text data associated with the chat session of the input query.

[0053] At 328, the computing system may utilize a machine learning embedding model to process the context-aware query to generate a query embedding. The query embedding may be associated with a plurality of values ​​determined based on processing the context-aware query. The query embedding may be generated based on features of the context-aware query associated with one or more topics and / or semantics of the query. The machine learning embedding model may be trained on a query intent pair (e.g., (query, intent)). The query intent may be associated with the intent of the corresponding query. The machine learning embedding model may process the query and intent of the query intent pair to generate a query embedding and an intent embedding. The loss function may be evaluated by comparing the intent embedding and the query embedding. One or more parameters of the machine learning embedding model may then be adjusted based on a gradient descent generated by the loss function based on a comparison of the query embedding and the intent embedding. The adjustment may be based on motivating the machine learning embedding model to generate similar embeddings for the query and the corresponding intent. In some implementations, the machine learning embedding model may be trained to generate similar embeddings for queries with similar query intents. For example, the model may be trained to map all queries "build your own smartphone case", "DIY phone case", and "make your own phone case" to the same embedding cluster because they have similar topics of homemade phone cases.

[0054] At 330, the computing system may determine a query embedding cluster associated with the query embedding. In some implementations, the query embedding cluster may be associated with multiple other embeddings associated with multiple other queries. For example, the query embedding may include the query "best deal smartphone cases," and the determined query embedding cluster may be "affordable smartphone cases," and include the query "cheap smartphone cases." Determining the query embedding cluster associated with the query embedding may include determining that the query embedding is associated with multiple other query embeddings within the learned distribution. The query embedding cluster determination may include determining that the context-aware query is associated with multiple other queries associated with nodes within the intent graph. In some implementations, the query embedding cluster may be associated with the learned intent graph. For example, the query embedding cluster may be a node within the learned intent graph that includes nodes of similar queries with boundaries connecting clusters with similar intents or attributes (e.g., topics, themes, etc.).

[0055] At 332, the computing system may determine multiple search results based on the query cluster. For example, the query cluster may include already known queries with already determined search results. Therefore, the system may generate multiple search results for the query cluster based on the queries within the query cluster and the already determined search results of the queries. In some implementations, determining multiple search results may include utilizing one or more search engines to process context-aware queries and / or one or more other queries associated with the query embedding cluster. Additionally and / or alternatively, determining multiple search results may include determining that one or more content items are associated with the query embedding cluster and / or the intent graph node. The one or more content items may be content items that are tagged and / or indexed using a specific cluster and / or a specific node index. Alternatively and / or additionally, search results may be determined based on cached search results associated with other queries in the query embedding cluster. In some implementations, a search engine may be utilized to process intent embedding and / or latent coding data associated with a node in the intent graph to determine multiple search results.

[0056] Additionally and / or alternatively, method 320 may include determining one or more attributes associated with the query embedding. Specifically, the system may determine attributes that may describe a particular topic associated with the input query or multiple rounds of query data. In some implementations, one or more attributes may be determined based on the query embedding cluster. For example, the system may determine the intent of the input query and one or more attributes of the intent (e.g., it is a "business-oriented" (e.g., the user is interested in purchasing something) intent or it is a "late purchase process" intent (e.g., the user is already quite far in the purchase process)). In another example, one or more attributes may be about the query itself (e.g., the query is a "niche" query or the topic of the query is an "economic" topic). In fact, there may be a variety of different attributes that the system can determine and utilize when determining search results.

[0057] Additionally and / or alternatively, method 320 may include determining a plurality of media content items based on the query embedding cluster. For example, the system may determine that the query embedding cluster is related to something such as being well suited for display via media (e.g., a picture or video). In some implementations, the media content items (e.g., a picture or video) may be determined along with or in lieu of the search results.

[0058] Figure 4 A block diagram of an example query rewrite system 400 for generating context-aware queries according to example aspects of the present disclosure is depicted. Figure 1 to Figure 2 , the system may include a generative language model 206 that receives as input an input query 202 and multiple rounds of query data 204 and may generate a context-aware query 208.

[0059] In one example, the input query may be example query 402. Example query 402 may be a simple question: "How much does it cost?" and nothing more. Example query 402 may be an accurate question, but "it" is unknown. Therefore, determining the subject of input query 202 may be difficult without additional information.

[0060] Additional information may be provided by the multi-pass query data 204 via the plurality of queries and responses 304. In one example, the plurality of messages (e.g., queries) and responses 304 may include an example user message 405 (e.g., a query and / or prompt) and an example response 404 (e.g., a chatbot response). For example, the example user message 405 may request information about a "new smartphone" through a series of messages. The example response 404 may provide information about the "new smartphone" based on the example user message 405. In some cases, the example query 402 may be part of the same query session as the multi-pass query data 204 and be from the query session.

[0061] The generative language model 206 can generate a context-aware query 208 based on the input query 202 and the additional information provided by the multi-turn query data 204. For example, the generative language model 206 can generate an example context-aware query 408 based on the example input query 402 and the example user message 405 and the example response 404 of the multi-turn query data 204. The generative language model can rewrite the example input query 402 to include the additional information from the multi-turn query data 204 by replacing the unknown “it” within the example input query 402 with the “new smartphone” topic discussed in the example user message 405 and the example response 404. In some implementations, and as depicted, the generative language model 206 can utilize information from both the user message and the responses of the plurality of user queries and responses 304 to determine the context-aware query 208.

[0062] Figure 5A block diagram of example components of a search result determination system 500 according to an example implementation of the present disclosure is depicted. The search result determination system 500 may include a graph data structure, such as an intent graph 308. In some implementations, the intent graph 308 may include multiple learning nodes, such as multiple query clusters 510. For example, the intent graph 308 may include multiple query clusters 510, each of which includes one or more queries that include relevant query intents. In one example, the query clusters may include queries such as "cheap phone cases", "best deal phone cases", and "affordable phone cases". In this example, the queries may be clustered based on the relevant query intents of low-priced and economical phone cases. As another example, the query clusters in the multiple query clusters 510 may include the queries "new phone cases", "popular phone cases", and "best new phone cases". In this example, the queries are clustered based on the relevant query intents of the latest and new phone cases.

[0063] In some implementations, intent graph 308 may include a plurality of boundaries 512 interconnecting a plurality of nodes, such as a plurality of query clusters 510. For example, plurality of boundaries 512 may connect nodes, such as having related node intents, such as query clusters having related query intents. For example, a boundary in plurality of boundaries 512 may connect a query cluster associated with low-priced and economical phone cases with a query cluster associated with latest and new phone cases. A boundary may represent related query intents between two query clusters (e.g., reasons why they need a smartphone case, or potential purchase intent they may have).

[0064] It should be understood that the example query clusters and boundaries depicted and discussed herein may be used for exemplary purposes only. In fact, intent graph 308 may include an unlimited number of query clusters and boundaries linking the clusters based on various topics and queries.

[0065] Figure 6A block diagram of an example embedding model training system 600 according to an example implementation of the present disclosure is depicted. The example embedding model training system 600 may include a first training pipeline 601 and a second training pipeline 602. The first training pipeline 601 and the second training pipeline 602 may be used to train a query embedding model 606 by back-propagating the output of a first loss function 612 and / or a second loss function 630 to the query embedding model 606, respectively. In some implementations, the first training pipeline 601 may be used to train the query embedding model 606, and the second training pipeline 602 may be used to fine-tune the query embedding model 606. In one implementation, the second training pipeline 602 may be used to train the query embedding model 606, and the first training pipeline 601 may be used to fine-tune the query embedding model 606. Alternatively and / or additionally, the first training pipeline 601 and / or the second training pipeline 602 may be executed in parallel and / or serially for training and / or fine-tuning.

[0066] The first training pipeline 601 can train and / or tune the query embedding model 606 by comparing the outputs (e.g., query embeddings) of multiple input queries via a first loss function 612. For example, the query embedding model 606 can receive a first input query 603 and a second input query 604 as input. The query embedding model 606 can generate a first query embedding 608 based on the first input query 603, and generate a second query embedding 610 based on the second input query 604. The first query embedding 608 and the second query embedding 610 can be used to evaluate the first loss function 612 to determine the loss between the two embeddings. The gradient descent determined based on the first loss function 612 can be back-propagated to the query embedding model 606 to adjust one or more parameters of the query embedding model 606 to train (and / or adjust) output generation. For example, two queries with similar intent can be input as the first input query 603 and the second input query 604. Their query embeddings (the first query embedding 608 and the second query embedding 610, respectively) can be input to the first loss function 612 to generate a loss. This loss may be back-propagated to the query embedding model 606 to train the embedding generation to output similar embeddings for queries with similar intent.

[0067] The second training pipeline 602 may train or tune the query embedding model 606 by comparing the output of the first input query 603 and the multi-round query data 624 via a second loss function 630. The multi-round query data 624 may include multiple queries and responses associated with the first input query 603 from the same query session. For example, a user may be involved in a chat-style query session, and their latest query is the first input query 603, and the multi-round query data 624 are the remaining queries and responses from the query session. In one implementation, the first input query 603 may be provided to the query embedding model 606 to generate a first query embedding 608, and input to the generative language model 626 along with the multi-round query data 624 to generate an intent determination 628. The intent determination may be a textual representation (and / or latent encoding) and / or a label (e.g., a classification label) of the intent of the first input query 603 determined by the generative language model 626 based on the first input query 603 and the multi-round query data 624. The first query embedding 608 and the intent determination 628 may be processed using a second loss function 630 to generate a loss, which may be back-propagated to the query embedding model 606 to train and / or tune the model. In some implementations, the intent determination 628 may be processed using the query embedding model 606 and / or other query embedding models trained in parallel with the query embedding model 606 to generate an intent embedding 629. The intent embedding 629 may be used to evaluate the second loss function 630 in conjunction with the first query embedding 608 in place of the intent determination 628 and / or in combination with the intent determination to generate a loss. The loss may then be back-propagated to the query embedding model 606 to train and / or fine-tune the model.

[0068] Figure 7 A block diagram of an example intent attribute determination system 700 according to an example implementation of the present disclosure is depicted. The example intent attribute determination system 700 may receive input data 702 including at least one of a context-aware query 208 or a query embedding 310. For example, the system 700 may receive the query embedding 310 as the input data 702. The input data 702 may be input to the generative language model 206 to generate one or more intent attributes 704. The intent attributes 704 may indicate one or more attributes associated with the determined intent of the input data 702. For example, the context-aware query 208 may be included in the input data 702, and the generated intent attributes 704 may be: the intent of the input data 702 (e.g., the context-aware query 208) is a "consumer-oriented" intent, it is a "late funnel" intent (e.g., the user has progressed to a considerable extent in terms of possibly purchasing an item), or it is a "niche question". In one implementation, the intent attributes 704 may be used to determine search results for a query associated with the input data 702, or stored for later use with a query associated with the input data 702.

[0069] Figure 8 A flow chart of an example method 800 for determining search results is depicted according to an example implementation of the present disclosure. Figure 8 The steps performed in a specific order are depicted for the purpose of illustration and discussion, but the method of the present disclosure is not limited to the specific illustrated order or arrangement. The various steps of method 800 may be omitted, rearranged, combined and / or adjusted in various ways without departing from the scope of the present disclosure.

[0070] At 802, the computing system may obtain an input query. For example, the computing system may receive the input query from a user via a chat-style query session. The user may provide multiple messages (e.g., multiple queries and / or multiple prompts) within a chat session (and / or a query session), and the input query is one of the multiple messages. In some implementations, the input query may include multimodal data, which includes two or more different types of data. For example, the input query may include text data as well as image data. In another implementation, the input query may include audio data, text data, and image data.

[0071] At 804, the computing system may obtain multi-turn query data. The multi-turn query data may describe previous inputs obtained before the query was input. In some cases, the previous inputs and the input query are associated with a particular multi-turn session (e.g., a chatty query session). For example, the computing system may obtain the entire chatty query session including the query and the response as the multi-turn query data. Specifically, the computing system may obtain a query from the chatty query session as the input query, and obtain the remaining data associated with the chatty query session as the multi-turn query data.

[0072] At 806, the computing system may utilize a machine learning language model to process the input query and the multi-round query data to generate a multi-round perception query. The multi-round perception query may describe the input query enhanced with additional details determined based on the multi-round query data. For example, the input query may include questions such as using pronouns, such as "How much does it cost?" The multi-round query data may include a history of queries and responses about the public topic "new smartphone X". The generative language model may process the input query and the multi-round query data (e.g., data describing the chat session history) to generate a multi-round perception query stating "How much does the new smartphone X cost?" This may include adding proper nouns from the multi-round query data to replace the pronouns from the input query. In some implementations, the machine learning language model may include a generative language model that is pre-trained on various content and text to perform multiple different language processing tasks. For example, the generative language model may be trained to process natural language input to perform text prediction, text correction, text enhancement, classification, sentiment analysis, semantic understanding, data enhancement, and / or other tasks. Therefore, in addition to generating multi-round perception queries, the machine learning language model is also capable of performing various tasks. In some implementations, the generative model can be a pre-trained machine learning language model.

[0073] At 808, the computing system may utilize a machine learning embedding model to process multiple rounds of perceptual queries to generate query embeddings. In some implementations, the query embedding model may be trained and / or tuned using the query and the generative model. For example, the query embedding model may be trained by generating a first query embedding for a first input query and a second query embedding for intent determination provided by the generative model. The generative model may generate an intent determination based on the first input query and the query session history. The first query embedding and the second query embedding may be processed by a loss function to generate a loss that may be back-propagated to the query embedding model to train the model. In one implementation, the intent determination may be input into a loss function along with the first query embedding to generate a loss that may be back-propagated to the query embedding model to train the model.

[0074] At 810, the computing system may determine a query embedding cluster associated with the query embedding. The query embedding cluster may be an embedded cluster associated with multiple different queries that have query intents similar to multi-round perception queries. The query embedding cluster may be associated with a node within a task graph, which is a data graph including multiple nodes associated with multiple different query tasks. In some implementations, the query embedding cluster may include multiple different queries associated with one or more shared attributes, and the one or more shared attributes may be associated with one or more query intents. For example, the query embedding cluster may include multiple different queries with shared attributes (e.g., having the same topic "smartphone case", having a similar intent "buy smart phone case" and / or having the same type of intent "consumer-oriented intent" and / or "later purchase intent").

[0075] At 812, the computing system may determine multiple search results based on the query cluster. For example, assuming that the input query provided is an informational question, or is linked to a requested resource or project, the computing system may determine an accurate answer to the input query. Generally speaking, the computing system may determine an accurate response to the input query based on the query embedding cluster associated with the query embedding. In some implementations, the multiple search results may include multimodal data, so that the multiple search results include at least two or more types of data. For example, the multiple search results may include text data and image data. In another example, the multiple search results may include text data, image data, and audio data.

[0076] Fig. 9 A flow chart is depicted for an example method 900 performed in accordance with an example implementation of the present disclosure. Fig. 9 The steps performed in a specific order are depicted for the purpose of illustration and discussion, but the method of the present disclosure is not limited to the specific illustrated order or arrangement. The various steps of method 900 may be omitted, rearranged, combined and / or adjusted in various ways without departing from the scope of the present disclosure.

[0077] At 902, a computing system may obtain input data. The input data may include a query. In some implementations, the query may be a rewritten query. For example, the query may be a rewritten query generated by obtaining an input query and multi-turn query data, and processing the input query and the multi-turn query data using a language model to generate the rewritten query. In some implementations, the multi-turn query data may describe previous input obtained before the input query and the previous input. In one implementation, the previous input and the input query may be associated with a specific multi-turn session.

[0078] At 904, the computing system may process the input data using the embedding model to generate a query embedding. The query embedding may describe mapping the input data to an embedding space. The query embedding may include multiple values ​​associated with features in the input data.

[0079] At 906, the computing system may utilize the generative model to process the input data to determine the query intent. The query intent may describe the type of information requested. The query intent may be a classification label, a latent encoding, and / or a text output.

[0080] At 908, the computing system may obtain one or more auxiliary query embeddings associated with one or more second query intents based on the query intent. The one or more second query intents may be associated with the query intent of the input data. In some implementations, the one or more second query embeddings may be obtained from a query cluster of a data graph associated with multiple query clusters. In some implementations, the query intent and the one or more second query intents may be associated with one or more specific topics. The type of information described by the query intent may then include additional details associated with the one or more specific topics.

[0081] At 910, the computing system may evaluate a loss function that evaluates the difference between the query embedding and one or more second query embeddings. In one implementation, the computing system may evaluate a loss function to evaluate the difference between the query embedding and one or more second query intents.

[0082] At 912, the computing system may adjust one or more parameters of the embedding model based at least in part on the loss function.

[0083] In some implementations, method 900 may include additional steps. For example, method 900 may include determining an intent embedding associated with the query intent. Thus, one or more second query embeddings may be obtained based on the intent embedding. Method 900 may also include generating a reconstructed data graph of the query cluster based on the query embedding, the intent embedding, and the one or more second query embeddings. The reconstructed data graph of the query cluster may include one or more boundaries associated with tangential topics of the query intent.

[0084] Query rewriting and query intent understanding can be implemented in a chat interface (e.g., a chat interface that supports a large language model) to determine the context of a given input based on an entire chat session, and generate a query that reflects the context of the given input and the entire chat session (e.g., details can be added to the query based on previous messages in the chat session). Query rewriting can utilize a large language model (LLM) to enhance the query based on the determined intent or context of the query, which is dependent on or relative to other previously provided inputs.

[0085] In a chat session (e.g., a chat session provided via an AI chatbot interface), user input may reference previous responses or inputs, which makes it difficult to discern the intent behind a given message from the text of the message alone (e.g., "What's the price of this?" alone is ambiguous because the message may reference a discussion topic that has been previously mentioned in the chat session (e.g., new 2023 phones from brand X (e.g., model Y)). If the intent of a message is unclear and / or ambiguous, search engines and other data processing systems may not be able to accurately provide responses and / or may not be able to generate relevant content items for the message.

[0086] A generative language model may be utilized to rewrite queries and / or prompts associated with a chat session to generate detailed queries (and / or prompts) that include details determined based on previous messages in the chat session. Specifically, the language understanding and generation capabilities of the generative language model may be used to process a given message and a chat session, and generate a new message that includes the given message and additional information from the chat session to summarize the intent of the given message. Additionally, the generative model may be utilized to tune previously generated intent graphs and intent clusters by evaluating embedding clusters using intent determinations of the LLM to supplement previous embedding model training / tuning techniques. Specifically, a generative language model (e.g., an LLM) may process a chat session history to determine additional context for an obtained input query.

[0087] A chat session may include multiple messages, which may include multiple different pieces of information that may provide limited information when viewed individually. By generating multi-turn aware queries based on the chat session history, a detailed query (and / or prompt) may be generated that includes details from the entire chat session, which may be utilized to obtain customized search results and / or detailed model generated responses. Specifically, AI-enabled chatbots already provide an interface where users can conduct multi-turn conversations; however, at the end of the conversation, the user and / or the system may request that an action be performed. Query enhancement and intent mapping to query clusters may be used to determine chat-aware search results.

[0088] The systems and methods disclosed herein may include using a generative model (e.g., LLM) to rewrite each "chat-like query" into a more complete query by looking at a user session (e.g., a user chat session). Additionally and / or alternatively, the systems and methods may include using an intent graph (e.g., a (LLM-driven) task graph) and a dual encoder intent mapping model to map the rewritten query into a query intent space.

[0089] The systems and methods may include mechanisms for contextually understanding chat-like queries, which may be hooked into intent models to facilitate content item retrieval and / or scoring.

[0090] For example, when provided with a complete user query history in the same session, a prompt may be designed, determined, generated, and / or constructed to request a generative language model (e.g., LLM) to generate a rewritten query. In some implementations, the generative model may be trained as a query enhancement model that may be actively built through one or more efforts. For example, the system and method may include a context engine that supports a generative language model. Additionally and / or alternatively, the system and method may utilize a generalized generative model pre-trained on multiple downstream tasks.

[0091] Example context-aware query generation may include: context: <chat about latest brand X phone>; user's query: "how much does it cost?"; and rewritten query: "brand X model 17 pro max price".

[0092] Context-aware queries (e.g., rewritten queries) may also be used as input to complete query intent discrimination (Query Intent-DR) to map the query to an intent space in an intent graph (e.g., a task graph). In some implementations, the system and method may accurately represent the contextual intent of a user query in an intent representation that may be directly used in various content item retrieval systems. The content item retrieval system may include an advertisement retrieval system (e.g., using intent to retrieve relevant advertisements / keywords) and one or more advertisement auction models (e.g., using intent as a feature to represent queries, advertisements, and user states).

[0093] In some implementations, the system and method may include using a generative model (eg, LLM) to further refine a current intent model, such as an intent graph (eg, task graph / query intent representation).

[0094] Intent models may be constructed based on user behavior signals (e.g., clustering clusters with similar click distributions in a search interface). In some implementations, one or more intent models may include an "encoder LLM" model to discover relevant queries in parallel with click signals to train and / or tune the intent model.

[0095] For example, the systems and methods may include using the LLM to further provide richer properties about the learning intent space (e.g., task graph nodes and edges), such as business attributes of intent and / or next step intent discovery.

[0096] To help both users and content item providers, the system can be configured and / or trained to understand deeper nuances above the intent representation by determining and / or providing (commercial) attributes. For example, after the system maps the user's context-aware (e.g., rewritten) query "Brand X model 17 pro max price" to a corresponding intent cluster, the system can further provide attributes above the concepts around the nodes of the query embedded cluster (e.g., commercial intent, "towards consumer" (2C) intent, and / or in the "later user funnel" that the user may have considered purchasing). In addition, the system and method can even suggest potential next steps for the intent (e.g., a user researching the intent "Brand X model 17 pro max price" may also be interested in researching "Brand X warranty").

[0097] For example, attributes can be used for content item targeting purposes. With the help of LLM, the system can build prompts with few-shot examples and can ask the LLM whether the query intent is consumer-oriented or business-oriented.

[0098] Another example may include discovering long-term task intent neighbors. For example, the systems and methods may include designing prompts to ask a generative model (e.g., LLM) what the best next steps are given the current query intent. The generative model may be configured and / or trained to generate long-term and / or task-oriented neighbors, which may be helpful in helping the user to accelerate progress.

[0099] Fig. 10A A block diagram of an example computing system 100 that performs search result generation according to an example implementation of the present disclosure is depicted. System 100 includes a user computing system 102, a server computing system 130, and / or a third party computing system 150 that are communicatively coupled via a network 180.

[0100] The user computing system 102 may include any type of computing device, such as a personal computing device (e.g., a laptop or desktop computer), a mobile computing device (e.g., a smartphone or tablet computer), a game console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.

[0101] The user computing system 102 includes one or more processors 112 and a memory 114. The one or more processors 112 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be one processor or a plurality of processors operatively connected. The memory 114 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 114 may store data 116 and instructions 118 that are executed by the processor 112 to cause the user computing system 102 to perform operations.

[0102] In some implementations, the user computing system 102 may store or include one or more machine learning models 120. For example, the machine learning model 120 may be or may otherwise include various machine learning models, such as a neural network (e.g., a deep neural network) or other types of machine learning models, including nonlinear models and / or linear models. The neural network may include a feedforward neural network, a recurrent neural network (e.g., a long short-term memory recurrent neural network), a convolutional neural network, or other forms of neural networks.

[0103] In some implementations, one or more machine learning models 120 may be received from the server computing system 130 via the network 180, stored in the user computing device memory 114, and then used or otherwise implemented by the one or more processors 112. In some implementations, the user computing system 102 may implement multiple parallel instances of a single machine learning model 120 (e.g., to perform parallel machine learning model processing across multiple instances of input data and / or detected features).

[0104] More specifically, the one or more machine learning models 120 may include one or more detection models, one or more classification models, one or more segmentation models, one or more enhancement models, one or more generation models, one or more natural language processing models, one or more optical character recognition models, and / or one or more other machine learning models. The one or more machine learning models 120 may include one or more Transformer models. The one or more machine learning models 120 may include one or more neural radiation field models, one or more diffusion models, and / or one or more autoregressive language models.

[0105] One or more machine learning models 120 can be used to detect one or more object features. The detected object features can be classified and / or embedded. The classification and / or embedding can then be used to perform a search to determine one or more search results. Alternatively and / or additionally, one or more detected features can be used to determine an indicator to be provided (e.g., a user interface element indicating the detected feature) to indicate that the feature has been detected. The user can then select the indicator to perform feature classification, embedding, and / or searching. In some implementations, the classification, embedding, and / or searching can be performed before selecting the indicator.

[0106] In some implementations, the one or more machine learning models 120 may process image data, text data, audio data, and / or potentially coded data to generate output data, which may include image data, text data, audio data, and / or potentially coded data. The one or more machine learning models 120 may perform optical character recognition, natural language processing, image classification, object classification, text classification, audio classification, context determination, action prediction, image correction, image enhancement, text enhancement, sentiment analysis, object detection, error detection, restoration, video stabilization, audio correction, audio enhancement, and / or data segmentation (e.g., mask-based segmentation).

[0107] Additionally or alternatively, one or more machine learning models 140 may be included in or otherwise stored and implemented by a server computing system 130 that communicates with the user computing system 102 according to a client-server relationship. For example, the machine learning model 140 may be implemented by the server computing system 130 as part of a web service (e.g., a viewfinder service, a visual search service, an image processing service, an environmental computing service, and / or an overlay application service). Thus, one or more models 120 may be stored and implemented at the user computing system 102, and / or one or more models 140 may be stored and implemented at the server computing system 130.

[0108] The user computing system 102 may also include one or more user input components 122 that receive user input. For example, the user input component 122 may be a touch-sensitive component (e.g., a touch-sensitive display screen or a touchpad) that is sensitive to the touch of a user input object (e.g., a finger or a stylus). The touch-sensitive component may be used to implement a virtual keyboard. Other example user input components include a microphone, a traditional keyboard, or other devices by which a user can provide user input.

[0109] In some implementations, the user computing system may store and / or provide one or more user interfaces 124, which may be associated with one or more applications. One or more user interfaces 124 may be configured to receive input and / or provide data for display (e.g., image data, text data, audio data, one or more user interface elements, augmented reality experience, virtual reality experience, and / or other data for display). The user interface 124 may be associated with one or more other computing systems (e.g., server computing system 130 and / or third-party computing system 150). The user interface 124 may include a viewfinder interface, a search interface, a generative model interface, a social media interface, and / or a media content library interface.

[0110] The user computing system 102 may include one or more sensors 126 and / or receive data from the one or more sensors. The one or more sensors 126 may be housed in a housing assembly that houses one or more processors 112, memory 114, and / or one or more hardware components that may store one or more software packages and / or cause the execution of one or more software packages. The one or more sensors 126 may include one or more image sensors (e.g., cameras), one or more lidar sensors, one or more audio sensors (e.g., microphones), one or more inertial sensors (e.g., inertial measurement units), one or more biosensors (e.g., heart rate sensors, pulse sensors, retinal sensors, and / or fingerprint sensors), one or more infrared sensors, one or more position sensors (e.g., GPS), one or more touch sensors (e.g., conductive touch sensors and / or mechanical touch sensors), and / or one or more other sensors. One or more sensors may be used to obtain data associated with a user environment (e.g., an image of the user environment, a record of the environment, and / or the user's location).

[0111] The user computing system 102 may include a user computing device 104 and / or be part of the user computing device. The user computing device 104 may include a mobile computing device (e.g., a smartphone or tablet computer), a desktop computer, a laptop computer, a smart wearable device, and / or a smart appliance. Additionally and / or alternatively, the user computing system may obtain data from one or more of the one or more user computing devices 104 and / or generate data using the one or more user computing devices. For example, a smartphone camera may be used to capture image data describing the environment, and / or an overlay application of the user computing device 104 may be used to track and / or process data provided to the user. Similarly, one or more sensors associated with a smart wearable device may be used to obtain data about the user and / or about the user's environment (e.g., a camera housed in the user's smart glasses may be used to obtain image data). Additionally and / or alternatively, data may be obtained and uploaded from other user devices that may be specifically used for data acquisition or generation.

[0112] The server computing system 130 includes one or more processors 132 and memory 134. The one or more processors 132 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be one processor or a plurality of processors operatively connected. The memory 134 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 134 may store data 136 and instructions 138 executed by the processor 132 to cause the server computing system 130 to perform operations.

[0113] In some implementations, the server computing system 130 includes or is otherwise implemented by one or more server computing devices. In instances where the server computing system 130 includes multiple server computing devices, such server computing devices may operate according to a sequential computing architecture, a parallel computing architecture, or some combination thereof.

[0114] As described above, the server computing system 130 may store or otherwise include one or more machine learning models 140. For example, the model 140 may be or may otherwise include various machine learning models. Example machine learning models include neural networks or other multi-layer nonlinear models. Example neural networks include feedforward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks. Fig. 9 Example model 140 is discussed.

[0115] Additionally and / or alternatively, server computing system 130 may include and / or be communicatively connected to a search engine 142 that may be used to crawl one or more databases (and / or resources). Search engine 142 may process data from user computing system 102, server computing system 130, and / or third-party computing system 150 to determine one or more search results associated with the input data. Search engine 142 may perform term-based searches, tag-based searches, Boolean-based searches, image searches, embedding-based searches (e.g., nearest neighbor searches), multimodal searches, and / or one or more other search techniques.

[0116] The server computing system 130 may store and / or provide one or more user interfaces 144 for obtaining input data and / or providing output data to one or more users. The one or more user interfaces 144 may include one or more user interface elements, which may include input fields, navigation tools, content widgets, selectable tiles, widgets, data display carousels, dynamic animations, information pop-ups, image enhancements, text-to-speech, speech-to-text, augmented reality, virtual reality, feedback loops, and / or other interface elements.

[0117] The user computing system 102 and / or the server computing system 130 may train the models 120 and / or 140 via interaction with a third party computing system 150 that is communicatively coupled via a network 180. The third party computing system 150 may be separate from the server computing system 130 or may be a part of the server computing system 130. Alternatively and / or additionally, the third party computing system 150 may be associated with one or more network resources, one or more network platforms, one or more other users, and / or one or more contexts.

[0118] The third-party computing system 150 may include one or more processors 152 and a memory 154. The one or more processors 152 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be one processor or multiple processors operatively connected. The memory 154 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc. and combinations thereof. The memory 154 may store data 156 and instructions 158 that are executed by the processor 152 to cause the third-party computing system 150 to perform operations. In some implementations, the third-party computing system 150 includes or is otherwise implemented by one or more server computing devices.

[0119] The network 180 may be any type of communication network, such as a local area network (e.g., an intranet), a wide area network (e.g., the Internet), or some combination thereof, and may include any number of wired or wireless links. In general, communications over the network 180 may be conducted over any type of wired and / or wireless connection using a variety of communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and / or protection schemes (e.g., VPN, secure HTTP, SSL).

[0120] The machine learning models described in this specification can be used for a variety of tasks, applications, and / or use cases.

[0121] In some implementations, the input to the machine learning model of the present disclosure may be image data. The machine learning model may process the image data to generate an output. As an example, the machine learning model may process the image data to generate an image recognition output (e.g., recognition of the image data, potential embedding of the image data, encoded representation of the image data, hashing of the image data, etc.). As another example, the machine learning model may process the image data to generate an image segmentation output. As another example, the machine learning model may process the image data to generate an image classification output. As another example, the machine learning model may process the image data to generate an image data modification output (e.g., a change to the image data, etc.). As another example, the machine learning model may process the image data to generate an encoded image data output (e.g., an encoded and / or compressed representation of the image data, etc.). As another example, the machine learning model may process the image data to generate an amplified image data output. As another example, the machine learning model may process the image data to generate a prediction output.

[0122] In some implementations, the input to the machine learning model of the present disclosure may be text or natural language data. The machine learning model may process the text or natural language data to generate an output. As an example, the machine learning model may process the natural language data to generate a language encoding output. As another example, the machine learning model may process the text or natural language data to generate a potential text embedding output. As another example, the machine learning model may process the text or natural language data to generate a translation output. As another example, the machine learning model may process the text or natural language data to generate a classification output. As another example, the machine learning model may process the text or natural language data to generate a text segmentation output. As another example, the machine learning model may process the text or natural language data to generate a semantic intent output. As another example, the machine learning model may process the text or natural language data to generate an amplified text or natural language output (e.g., text or natural language data of higher quality than the input text or natural language, etc.). As another example, the machine learning model may process the text or natural language data to generate a prediction output.

[0123] In some implementations, the input to the machine learning model of the present disclosure may be speech data. The machine learning model may process the speech data to generate an output. As an example, the machine learning model may process the speech data to generate a speech recognition output. As another example, the machine learning model may process the speech data to generate a speech translation output. As another example, the machine learning model may process the speech data to generate a potential embedding output. As another example, the machine learning model may process the speech data to generate an encoded speech output (e.g., an encoded and / or compressed representation of the speech data, etc.). As another example, the machine learning model may process the speech data to generate an amplified speech output (e.g., speech data of higher quality than the input speech data, etc.). As another example, the machine learning model may process the speech data to generate a text representation output (e.g., a text representation of the input speech data, etc.). As another example, the machine learning model may process the speech data to generate a predicted output.

[0124] In some implementations, the input to the machine learning model of the present disclosure may be sensor data. The machine learning model may process the sensor data to generate an output. As an example, the machine learning model may process the sensor data to generate a recognition output. As another example, the machine learning model may process the sensor data to generate a prediction output. As another example, the machine learning model may process the sensor data to generate a classification output. As another example, the machine learning model may process the sensor data to generate a segmentation output. As another example, the machine learning model may process the sensor data to generate a segmentation output. As another example, the machine learning model may process the sensor data to generate a visualization output. As another example, the machine learning model may process the sensor data to generate a diagnostic output. As another example, the machine learning model may process the sensor data to generate a detection output.

[0125] In some cases, the input includes visual data, and the task is a computer vision task. In some cases, the input includes pixel data for one or more images, and the task is an image processing task. For example, the image processing task may be image classification, where the output is a set of scores, each score corresponding to a different object class and representing the possibility that one or more images depict an object belonging to the object class. The image processing task may be object detection, where the image processing output identifies one or more regions in one or more images, and for each region, identifies the possibility that the region depicts an object of interest. As another example, the image processing task may be image segmentation, where the image processing output defines, for each pixel in one or more images, a corresponding possibility for each category in a predetermined set of categories. For example, the set of categories may be foreground and background. As another example, the set of categories may be an object class. As another example, the image processing task may be depth estimation, where the image processing output defines a corresponding depth value for each pixel in one or more images. As another example, the image processing task may be motion estimation, where the network input includes multiple images, and the image processing output defines, for each pixel of one of the input images, the motion of the scene depicted at the pixel between images in the network input.

[0126] The user computing system may include multiple applications (e.g., applications 1 to N). Each application may include its own corresponding machine learning library and machine learning model. For example, each application may include a machine learning model. Example applications include text messaging applications, email applications, dictation applications, virtual keyboard applications, browser applications, etc.

[0127] Each application can communicate with multiple other components of a computing device, such as one or more sensors, a context manager, a device state component, and / or additional components. In some implementations, each application can communicate with each device component using an API (e.g., a public API). In some implementations, the API used by each application is specific to the application.

[0128] The user computing system 102 may include multiple applications (e.g., applications 1 to N). Each application communicates with a central intelligence layer. Example applications include text messaging applications, email applications, dictation applications, virtual keyboard applications, browser applications, etc. In some implementations, each application may communicate with the central intelligence layer (and the models stored therein) using an API (e.g., a public API across all applications).

[0129] The central intelligence layer may include multiple machine learning models. For example, a corresponding machine learning model (e.g., model) may be provided for each application and managed by the central intelligence layer. In other implementations, two or more applications may share a single machine learning model. For example, in some implementations, the central intelligence layer may provide a single model (e.g., a single model) for all applications. In some implementations, the central intelligence layer is included within or otherwise implemented by an operating system of computing system 100.

[0130] The central intelligence layer can communicate with a central device data layer. The central device data layer can be a centralized data repository for the computing system 100. The central device data layer can communicate with multiple other components such as a computing device (e.g., one or more sensors, context managers, device state components, and / or additional components). In some implementations, the central device data layer can communicate with each device component using an API (e.g., a private API).

[0131] Fig. 10BA block diagram of an example computing system 50 that performs search result generation according to an example implementation of the present disclosure is depicted. Specifically, the example computing system 50 may include one or more computing devices 52, which may be used to obtain and / or generate one or more data sets, which may be processed by a sensor processing system 60 and / or an output determination system 80 for feedback to a user, who may provide information about features in one or more obtained data sets. The one or more data sets may include image data, text data, audio data, multimodal data, potentially encoded data, etc. The one or more data sets may be obtained via one or more sensors associated with the one or more computing devices 52 (e.g., one or more sensors in the computing devices 52). Additionally and / or alternatively, the one or more data sets may be stored data and / or retrieved data (e.g., data retrieved from a network resource). For example, a user may interact with images, text, and / or other content items. Interaction with the content items may then be utilized to generate one or more determinations.

[0132] One or more computing devices 52 may obtain and / or generate one or more data sets based on image capture, sensor tracking, data storage retrieval, content download (e.g., downloading images or other content items from web resources via the Internet), and / or via one or more other technologies. One or more data sets may be processed using a sensor processing system 60. The sensor processing system 60 may perform one or more processing techniques using one or more machine learning models, one or more search engines, and / or one or more other processing techniques. One or more processing techniques may be performed in any combination and / or individually. One or more processing techniques may be performed serially and / or in parallel. Specifically, one or more data sets may be processed using a context determination block 62, which may determine a context associated with one or more content items. The context determination block 62 may identify and / or process metadata, user profile data (e.g., preferences, user search history, user browsing history, user purchase history, and / or user input data), previous interaction data, global trend data, location data, time data, and / or other data to determine a specific context associated with a user. The context may be associated with an event, a determined trend, a particular action, a particular type of data, a particular environment, and / or another context associated with a user and / or retrieved or obtained data.

[0133] The sensor processing system 60 may include an image pre-processing block 64. The image pre-processing block 64 may be used to adjust one or more values ​​of an acquired and / or received image to prepare the image to be processed by one or more machine learning models and / or one or more search engines 74. The image pre-processing block 64 may resize the image, adjust saturation values, adjust resolution, strip and / or add metadata, and / or perform one or more other operations.

[0134] In some implementations, the sensor processing system 60 may include one or more machine learning models, which may include a detection model 66, a segmentation model 68, a classification model 70, an embedding model 72, and / or one or more other machine learning models. For example, the sensor processing system 60 may include one or more detection models 66 that may be used to detect specific features in a processed dataset. Specifically, one or more detection models 66 may be used to process one or more images to generate one or more bounding boxes associated with detected features in the one or more images.

[0135] Additionally and / or alternatively, the one or more segmentation models 68 may be used to segment one or more portions of a data set from one or more data sets. For example, the one or more segmentation models 68 may utilize one or more segmentation masks (e.g., one or more segmentation masks generated manually and / or based on one or more bounding boxes) to segment a portion of an image, a portion of an audio file, and / or a portion of text. Segmentation may include isolating one or more detected objects and / or removing one or more detected objects from an image.

[0136] One or more classification models 70 may be used to process image data, text data, audio data, potentially encoded data, multimodal data, and / or other data to generate one or more classifications. One or more classification models 70 may include one or more image classification models, one or more object classification models, one or more text classification models, one or more audio classification models, and / or one or more other classification models. One or more classification models 70 may process data to determine one or more classifications.

[0137] In some implementations, one or more embedding models 72 may be used to process data to generate one or more embeddings. For example, one or more embedding models 72 may be used to process one or more images to generate one or more image embeddings in an embedding space. The one or more image embeddings may be associated with one or more image features of the one or more images. In some implementations, the one or more embedding models 72 may be configured to process multimodal data to generate multimodal embeddings. The one or more embeddings may be used to classify, search, and / or learn an embedding space distribution.

[0138] The sensor processing system 60 may include one or more search engines 74 that can be used to perform one or more searches. The one or more search engines 74 can crawl one or more databases (e.g., one or more local databases, one or more global databases, one or more private databases, one or more public databases, one or more dedicated databases, and / or one or more general databases) to determine one or more search results. The one or more search engines 74 can perform feature matching, text-based searches, embedding-based searches (e.g., k-nearest neighbor searches), metadata-based searches, multimodal searches, web resource searches, image searches, text searches, and / or application searches.

[0139] Additionally and / or alternatively, the sensor processing system 60 may include one or more multimodal processing blocks 76 that may be used to assist in processing multimodal data. The one or more multimodal processing blocks 76 may include generating multimodal queries and / or multimodal embeddings for processing by one or more machine learning models and / or one or more search engines 74.

[0140] The output of the sensor processing system 60 may then be processed using an output determination system 80 to determine one or more outputs to be provided to the user. The output determination system 80 may include a heuristics-based determination, a machine learning model-based determination, a user selection-based determination, and / or a context-based determination.

[0141] The output determination system 80 can determine how and / or where to provide one or more search results in the search results interface 82. Additionally and / or alternatively, the output determination system 80 can determine how and / or where to provide one or more machine learning model outputs in the machine learning model output interface 84. In some implementations, one or more search results and / or one or more machine learning model outputs can be provided for display via one or more user interface elements. One or more user interface elements can be superimposed on the displayed data. For example, one or more detection indicators can be superimposed on the detected object in the viewfinder. One or more user interface elements can be selected to perform one or more additional searches and / or one or more additional machine learning model processes. In some implementations, user interface elements can be provided as dedicated user interface elements for a specific application and / or can be uniformly provided across different applications. One or more user interface elements may include pop-up displays, interface overlays, interface tiles and / or small pieces, carousel interfaces, audio feedback, animations, interactive widgets, and / or other user interface elements.

[0142] Additionally and / or alternatively, data associated with the output of the sensor processing system 60 may be utilized to generate and / or provide an augmented reality experience and / or a virtual reality experience 86. For example, one or more acquired data sets may be processed to generate one or more augmented reality rendering assets and / or one or more virtual reality rendering assets, which may then be used to provide an augmented reality experience and / or a virtual reality experience 86 to a user. The augmented reality experience may render information associated with an environment into a corresponding environment. Alternatively and / or additionally, objects associated with the processed data sets may be presented to a user environment and / or a virtual environment. Rendering data set generation may include training one or more neural radiance field models to learn a three-dimensional representation of one or more objects.

[0143] In some implementations, one or more action prompts 88 may be determined based on the output of the sensor processing system 60. For example, a search prompt, a purchase prompt, a generate prompt, a reservation prompt, a call prompt, a redirect prompt, and / or one or more other prompts may be determined to be associated with the output of the sensor processing system 60. The one or more action prompts 88 may then be provided to the user via one or more selectable user interface elements. In response to selection of the one or more selectable user interface elements, a corresponding action of the corresponding action prompt may be performed (e.g., a search may be performed, a purchase application programming interface may be utilized, and / or another application may be opened).

[0144] In some implementations, one or more data sets and / or outputs of the sensor processing system 60 may be processed using one or more generative models 90 to generate model-generated content items that may then be provided to a user. The generation may be prompted based on a user selection and / or may be performed automatically (e.g., automatically based on one or more conditions, which may be associated with a threshold amount of search results that are not represented).

[0145] One or more generative models 90 may include a language model (e.g., a large language model and / or a visual language model), an image generation model (e.g., a text-to-image generation model and / or an image enhancement model), an audio generation model, a video generation model, a graphics generation model, and / or other data generation models (e.g., other content generation models). One or more generative models 90 may include one or more Transformer models, one or more convolutional neural networks, one or more recurrent neural networks, one or more feedforward neural networks, one or more generative adversarial networks, one or more self-attention models, one or more embedding models, one or more encoders, one or more decoders, and / or one or more other models. In some implementations, one or more generative models 90 may include one or more autoregressive models (e.g., a machine learning model trained to generate predicted values ​​based on prior behavior data) and / or one or more diffusion models (e.g., a machine learning model trained to generate predicted data based on generating and processing distribution data associated with input data).

[0146] One or more generative models 90 can be trained to process input data and generate model-generated content items, which can include multiple predicted words, pixels, signals and / or other data. Model-generated content items can include novel content items that are different from any existing works. One or more generative models 90 can utilize learned representations, sequences and / or probability distributions to generate content items, which can include phrases, storylines, settings, objects, characters, beats, lyrics, and / or other aspects not included in existing content items.

[0147] The one or more generative models 90 may include a visual-linguistic model.

[0148] The visual language model can be trained, tuned, and / or configured to process image data and / or text data to generate natural language output. The visual language model can utilize a pre-trained large language model (e.g., a large autoregressive language model) and one or more encoders (e.g., one or more image encoders and / or one or more text encoders) to provide detailed natural language output that simulates natural language written by humans.

[0149] The visual language model can be used for zero-shot image classification, few-shot image classification, image captioning, multimodal query refinement, multimodal question answering, and / or can be tuned and / or trained for multiple different tasks. The visual language model can perform visual question answering, image caption generation, feature detection (e.g., content monitoring (e.g., for inappropriate content)), object detection, scene recognition, and / or other tasks.

[0150] The visual language model can utilize a pre-trained language model and then tune it to achieve multimodality. The training and / or tuning of the visual language model can include image-text matching, masked language modeling, multimodal fusion of cross attention, contrastive learning, prefix language model training, and / or other training techniques. For example, the visual language model can be trained to process images to generate predicted text similar to real-value text data (e.g., real-value subtitles of images). In some implementations, the visual language model can be trained to replace the masked word units of the natural language template with text word units describing the features depicted in the input image. Alternatively and / or additionally, the training, tuning and / or model inference can include a multi-layer cascade of visual and text embedding features. In some implementations, the visual language model can be trained and / or tuned by jointly learning image embedding and text embedding generation, which can include training and / or tuning the system to map the embedding to a joint feature embedding space that maps text features and image features to a shared embedding space. Joint training can include parallel embedding of image-text pairs and / or can include triple training. In some implementations, images can be used and / or processed as prefixes for language models.

[0151] The output determination system 80 may utilize the data enhancement block 92 to process one or more data sets and / or the output of the sensor processing system 60 to generate enhanced data. For example, the data enhancement block 92 may be utilized to process one or more images to generate one or more enhanced images. Data enhancement may include data correction, data cropping, removal of one or more features, addition of one or more features, resolution adjustment, lighting adjustment, saturation adjustment, and / or other enhancements.

[0152] In some implementations, one or more data sets and / or outputs of the sensor processing system 60 may be stored based on the determination of the data storage block 94 .

[0153] The output of the output determination system 80 may then be provided to the user via one or more output components of the user computing device 52. For example, one or more user interface elements associated with the one or more outputs may be provided for display via a visual display of the user computing device 52.

[0154] The process may be performed iteratively and / or continuously. One or more user inputs to the provided user interface elements may modulate and / or influence the continuous processing loop.

[0155] The technology discussed herein relates to servers, databases, software applications, and other computer-based systems and the actions taken and the information sent to and from these systems. The inherent flexibility of computer-based systems allows for multiple possible configurations, combinations, and partitioning of tasks and functions between and among components. For example, the processes discussed herein can be implemented using a single device or component or multiple devices or components working in combination. Databases and applications can be implemented on a single system or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0156] Although the subject matter has been described in detail with respect to various specific example embodiments of the subject matter, each example is provided by way of explanation, rather than limitation of the present disclosure. Those skilled in the art, after understanding the foregoing, may easily produce changes, modifications, and equivalents to these embodiments. Therefore, the present disclosure does not exclude such modifications, changes, and / or additions to the subject matter that would be readily understood by those of ordinary skill in the art. For example, a feature shown or described as part of one embodiment may be used together with another embodiment to produce yet another embodiment. Therefore, the present disclosure is intended to cover these changes, modifications, and equivalents.

Claims

1. A computer-implemented method for generating a context-aware query, the method comprising: obtaining, by a computing system including one or more computing devices, an input query; obtaining, by the computing system, multi-turn query data, wherein the multi-turn query data describes previous input obtained prior to the input query, wherein the previous input and the input query are associated with a particular multi-turn session; processing, by the computing system, the input query and the multiple rounds of query data to generate the context-aware query, wherein the context-aware query describes the input query and additional details, wherein the additional details describe a context of the input query associated with the multiple rounds of query data; Processing the context-aware query using a machine learning embedding model by the computing system to generate a query embedding; determining, by the computing system, a query embedding cluster associated with the query embedding, wherein the query embedding cluster is associated with a plurality of other embeddings associated with a plurality of other queries; as well as A plurality of search results is determined, by the computing system, based on the query embedding cluster.

2. The computer-implemented method of claim 1, wherein the machine learning embedding model is trained to generate embeddings that map embeddings associated with similar query intents to a shared embedding cluster.

3. The computer-implemented method of claim 1 , further comprising: One or more attributes associated with the query embedding are determined, by the computing system and based on the query embedding cluster, wherein the one or more attributes describe a specific topic associated with at least one of the input query or the plurality of query data.

4. A computer-implemented method as described in claim 1, wherein the query embedding is associated with a query intent pair comprising the context-aware query and a query intent, wherein the query intent is associated with the intent of the input query and the context-aware query.

5. The computer-implemented method of claim 4, wherein determining a query embedding cluster associated with the query embedding comprises: Mapping the query intent pair to an embedding space by the computing system; as well as The query-intent pair is determined, by the computing system and based on at least one of the query embedding or the intent embedding, to be associated with a plurality of other embeddings associated with nodes within a query graph.

6. The computer-implemented method of claim 1 , wherein the query embedding cluster is associated with a learned intent graph, wherein the learned intent graph comprises: a plurality of nodes, wherein each node represents a cluster of queries having related query intent; as well as A plurality of edges, wherein the plurality of edges connect nodes having related node intents.

7. The computer-implemented method of claim 1 , further comprising: A plurality of media content items are determined, by the computing system, based on the query embedding cluster.

8. The computer-implemented method of claim 1, wherein the input query comprises multimodal data, wherein the multimodal data comprises two or more different types of data.

9. The computer-implemented method of claim 8, wherein the multimodal data comprises image data and text data, and wherein the context-aware query is generated based on the image data, the text data, and the multi-turn query data.

10. A computing system for generating a multi-round perception query, the system comprising: one or more processors; as well as one or more non-transitory computer-readable media collectively storing instructions that, when executed by the one or more processors, cause the computing system to perform operations comprising: Get input query; obtaining multi-turn query data, wherein the multi-turn query data describes previous input obtained prior to the input query, wherein the previous input and the input query are associated with a particular multi-turn session; Processing the input query and the multi-pass query data using a machine learning language model to generate the multi-pass aware query, wherein the multi-pass aware query describes the input query enhanced with additional details determined based on the multi-pass query data; Processing the plurality of rounds of perception queries using a machine learning embedding model to generate query embeddings; determining a query embedding cluster associated with the query embedding; and A plurality of search results are determined based on the query embedding cluster.

11. The computing system of claim 10, wherein the machine learning language model comprises a generative language model that is pre-trained on a variety of content and text to perform a plurality of different language processing tasks.

12. A computing system as described in claim 10, wherein the query embedding cluster is an embedded cluster associated with multiple different queries, the multiple different queries have similar query intents as the multi-round perception query, and wherein the query embedding cluster is associated with a node within a task graph, wherein the task graph includes multiple learning nodes associated with multiple different query tasks.

13. The computing system of claim 10, wherein the query embedding cluster further comprises a plurality of different queries associated with one or more shared attributes, wherein the one or more shared attributes are associated with one or more query intents.

14. One or more non-transitory computer-readable media collectively storing instructions that, when executed by a computing system, cause the computing system to perform operations comprising: obtaining input data, wherein the input data comprises a query; Processing the query using an embedding model to generate a query embedding; Processing the input data using a generative model to determine query intent, wherein the query intent describes a type of information requested; obtaining one or more second query embeddings associated with one or more second queries having one or more second query intents based on the query intent, wherein the one or more second query intents are associated with the query intent of the input data; evaluating a loss function that evaluates a difference between the query embedding and the one or more second query embeddings; as well as One or more parameters of the embedding model are adjusted based at least in part on the loss function.

15. The non-transitory computer-readable medium of claim 14, wherein the query comprises a rewritten query, wherein the rewritten query is generated by: obtaining an input query and multi-turn query data, wherein the multi-turn query data describes previous input obtained prior to the input query, wherein the previous input and the input query are associated with a particular multi-turn session; and The input query and the multiple rounds of query data are processed using a language model to generate the rewritten query.

16. The non-transitory computer-readable medium of claim 14, wherein the one or more second query embeddings are obtained from a query cluster of a data graph associated with a plurality of query clusters.

17. The non-transitory computer readable medium of claim 14, wherein the operations further comprise: Determining that an intent embedding is associated with the query intent; and Wherein the one or more second query embeddings are obtained based on the intent embedding.

18. The non-transitory computer-readable medium of claim 14, wherein the query intent and the one or more second query intents are associated with one or more specific topics, and wherein the type of information includes additional details associated with the one or more specific topics.

19. The non-transitory computer readable medium of claim 14, wherein the operations further comprise: A reconstructed data graph of query clusters is generated based on the query embeddings, the intent, and the one or more second query embeddings.

20. The non-transitory computer-readable medium of claim 19, wherein the reconstructed data graph of query clusters includes one or more boundaries associated with tangential topics of query intent.

Images