Multimodal query processing for audio interfaces

Abstract

Techniques are provided for multimodal query processing for audio-only interface devices. Embodiments receive an audio query from a user and translate the audio query to a text-based query input. A multimodal LLM processes the text-based query input to return multimodal query responses and a query context. A modality projection engine classifies each multimodal query result into an appropriate modality, uses specifically tailored modality-to-text converters to convert each of the multimodal query responses to a corresponding text-based response, summarizes the text-based responses based on the query context to generate a text-based query output, and synthesizes and returns an audio query response from the text-based query output.

Description

MULTIMODAL QUERY PROCESSING FOR AUDIO INTERFACESBACKGROUND

[0001] In recent years, there has been a proliferation of smart speaker devices in people’s homes, offices, and other locations. Such devices typically provide an audio-only interface by which a user can issue voice commands or prompts to trigger a corresponding action or receive an appropriate reply (i.e., such devices may not include a display). For example, smart speakers can be integrated with a smart home mesh and used as an interface to control lighting, appliances, and / or other smart home devices. In some cases, smart speakers can be in communication with the Internet (e.g., via a local Internet router) and can be used as an interface for initiating web-based queries. For example, a user verbally asks a query to the smart speaker, the query is converted to text, the text is used to perform an Internet search and to receive a response, and the response is converted back to audio and played back to the user.

[0002] Although many older audio-only interface devices restricted users to a limited set of predefined audio commands, modem audio-only interface devices typically seek to support natural language queries. Processing of such natural language queries can depend on large language models (LLMs) and related applications. LLMs are artificial intelligence models that are trained on massive libraries of information to learn patterns and structures of language from which to generate appropriate outputs. For example, vast corpora of textual data are fed into the LLM, and deep learning, big data, and other artificial intelligence and machine learning techniques are used to train very large numbers (e.g., millions or billions) of model parameters. Most LLMs tend to be text-based LLMs; trained on textual datasets to provide textual outputs. However, some LLMs, referred to as multimodal LLMs, can be trained with datasets in several modalities, such as text, audio, images, videos, etc. For example, when generating an appropriate output to a query, a multimodal LLM may output an appropriate textual response, photograph, audio clip, illustration, etc.

[0003] While multimodal LLMs provide an opportunity for a wide range of query responses across multiple modalities, most of those modalities are not directly compatible with an audio-only interface. For example, a smart speaker device may not have any display by which to visually output an image or a video query response. Thus, conventionally, audio-only interface devices interface with such text-based LLMs. For example, a user verbally asks a query to the smart speaker, the query is converted to text, the text is used to perform an Internet search and to receive a text response, and the text response is converted back to audio and played back to the user.SUMMARY

[0004] Systems and methods described herein provide multimodal query processing for audio- only interface devices. Embodiments receive an audio query from a user and translate the audio query to a text-based query input. A multimodal LLM processes the text-based query input to return multimodal query responses and a query context. A modality projection engine classifies each multimodal query result into an appropriate modality, uses specifically tailored modality -to- text converters to convert each of the multimodal query responses to a corresponding text-based response, summarizes the text-based responses based on the query context to generate a text-based query output, and synthesizes and returns an audio query response from the text-based query output.

[0005] In one set of embodiments, a method is provided for multimodal query processing. The method includes: converting an audio query input signal to a text-based query prompt; processing the text-based query prompt by a multimodal query large language model (LLM) to generate a plurality of multimodal query results and query context data, each query result associated with one of a plurality of modalities; converting the plurality of multimodal query results to a plurality of text-based query results, such that each text-based query result corresponds to one of the plurality of multimodal query results; generating a text-based query output by concatenating and summarizing the plurality of text-based query results based on the query context data; and synthesizing an audio query output from the text-based query output.

[0006] In another set of embodiments, a system is provided for multimodal query processing. The system includes: a speech-to-text transcription engine to convert an audio query input signal to a text-based input signal; a query engineering engine to generate a text-based query prompt from the text-based input signal; a multimodal query large language model (LLM) to process the text-based query prompt to generate a plurality of multimodal query results and query context data, each multimodal query result associated with one of a plurality of modalities; a modality projection engine to: convert the plurality of multimodal query results to a plurality of text-based query results, such that each text-based query result corresponds to one of the plurality of multimodal query results; and generate a text-based query output by summarizing the plurality of text-based query results based on the query context data; and a text-to-speech rendering engine to synthesize an audio query output from the text-based query output.

[0007] In another set of embodiments, another system is provided. The system includes one or more processors; and one or more non-transitory, processor-readable memories having instructions stored. When executed, the instructions cause the one or more processors to implementcomponents of a system for multimodal query processing and / or to perform a method for multimodal query processing.BRIEF DESCRIPTION OF THE DRAWINGS

[0008] A further understanding of the nature and advantages of various embodiments may be realized by reference to the following figures. In the appended figures, similar components or features may have the same reference label. Further, various components of the same type may be distinguished by following the reference label by a dash and a second label that distinguishes among the similar components. If only the first reference label is used in the specification, the description is applicable to any one of the similar components having the same first reference label irrespective of the second reference label.

[0009] FIG. 1 shows an illustrative multimodal query processing environment, according to embodiments described herein.

[0010] FIG. 2 shows a block diagram of an illustrative embodiment of a partial back-end system for multimodal query processing, according to embodiments described herein.

[0011] FIG. 3 shows another approach for classifying the multimodal query results by the modality classifier in a partial back-end system.

[0012] FIG. 4 shows a partial modality projection engine, including a modality classifier and an illustrative modality -to-text converter.

[0013] FIG. 5 shows a flow diagram of an illustrative method for multimodal query processing in context of an audio-only interface device, according to embodiments described herein.

[0014] FIG. 6 shows an illustrative distributed computational environment for implementing multimodal query processing, according to embodiments described herein.DETAILED DESCRIPTION

[0015] A number of consumer electronic devices are available that provide consumers with an audio-only interface to Internet content. For example, many consumers have smart speaker devices in their homes or offices that provide audio responses to audio queries. Many such devices support natural language queries, which allow consumers to issue verbal queries or commands without being limited to a predefined list of audio commands. Processing of such natural language queries is typically performed by a large language model (LLM). For example, a verbal query is received by a smart speaker and converted to text, the text is processed by a textbased LLM to generate an appropriate query prompt and receive an appropriate text-based response, and the text-based response is converted back to audio and played back to the consumervia the smart speaker. Conventionally, query processing for audio-only interface devices has been restricted to text-based LLMs in order to restrict query responses to a textual modality that can easily be converted to an audio output compatible with the audio-only interface. For example, while multimodal LLMs can be used to generate responses in a variety of modalities (i.e., not only text), such a multimodal response framework has conventionally been considered incompatible with audio-only interfaces.

[0016] As described herein, embodiments provide multimodal query processing for audio-only interface devices. Embodiments receive an audio query from a user and translate the audio query to a text-based query input. A multimodal LLM processes the text-based query input to return multimodal query responses and a query context. A modality projection engine classifies each multimodal query result into an appropriate modality, uses specifically tailored modality -to-text converters to convert each of the multimodal query responses to a corresponding text-based response, summarizes the text-based responses based on the query context to generate a text-based query output, and synthesizes and returns an audio query response from the text-based query output.

[0017] “Modalities” refer to output data type categories, such as text-based modalities, audiobased modalities, image-based modalities, video-based modalities, and the like. Embodiments described herein provide multimodal query processing. As used herein, such “multimodal” processing includes support for at least one text-based modality and at least one non-text-non- audio modality (e.g., images, videos, etc.). Some embodiments described herein process modalities in a hierarchical manner. In such embodiments, each modality can be a highest-level category, and each of some or all of the modalities can have a set of “sub-modalities.” For example, an “image” modality can have several sub-modalities, such as “photograph,” “illustration,” “data plot,” “table,” “clipart,” “line drawing,” etc.

[0018] FIG. 1 shows an illustrative multimodal query processing environment 100, according to embodiments described herein. The environment 100 includes an audio-only interface device 120 in communication with a back-end system 130 via one or more communication networks 180. The audio-only interface device 120 can be a smart speaker, or any other suitable network-connected device with which a consumer 110 interacts via an audio-only interface. As illustrated, the audio- only interface device 120 receives an audio query input signal 105 from the consumer 110 via one or more microphones, and the audio-only interface device 120 returns an audio query output 107 to the consumer 110 via one or more speakers.

[0019] Embodiments of the back-end system 130 are implemented in the cloud. As used herein, the term “cloud” refers generally to a network of remote servers hosted on the Internet that provides services to consumers using distributed storage, computation, application, and other computing resources. As such, the network(s) 180 provide the audio-only interface device 120 at least with access to the cloud (i.e., to the Internet). Though not explicitly shown, the audio-only interface device 120 may be communicatively coupled with the network(s) 180 via one or more intermediate devices, such as via a network modem and / or router that provides wired and / or wireless connectivity to the audio-only interface device 120 via one or more local area networks (LANs). The network(s) 180 can generally include any type of wired or wireless network links, or combinations thereof.

[0020] Embodiments of the back-end system 130 include a speech-to-text transcription engine 125, a query engineering engine 140, a multimodal (MM) query large language model (qLLM) 150, a modality projection engine 160, and a text-to-speech rendering engine 170. The back-end system 130 can also include one or more network interfaces 175 for communicatively coupling with the network(s) 180. For example, the speech-to-text transcription engine 125 is coupled with the network interface 175 to receive the audio query input 105 from the audio-only interface device 120 via the network(s) 180 responsive to generation of the audio query input 105 based on a spoken query received from the consumer 110 via an audio-only user interface; and the text-to- speech rendering engine 170 is coupled with the network interface 175 to transmit the audio query output 107 to the audio-only interface device 120 via the network(s) 180 for output to the consumer 110 via the audio-only user interface.

[0021] Embodiments of the audio-only interface device 120 can process the audio query input signal 105 prior to sending it over the network(s) 180 to the back-end system 130 as a transmitted audio query input signal 105’. For example, the audio-only interface device 120 converts the received analog audio query input signal 105 into a digital representation of the audio data and further performs compressing, filtering, packetizing, and / or other processing of the signal to generate the transmitted audio query input signal 105’. Similarly, embodiments of the audio-only interface device 120 can process a received audio query output 107’ received via the network(s) 180 prior to outputting the signal as the audio query output 107 to the consumer 110. For example, the audio-only interface device 120 converts a compressed, packetized, and / or otherwise processed digital audio signal into an audio signal suitable for driving one or more speakers of the audio-only interface.

[0022] Although all of the query processing components are illustrated as part of the back-end system 130 implemented in the cloud, alternatively embodiments can implement one or more components in the audio-only interface device 120. For example, the speech-to-text transcription engine 125 and / or the text-to-speech rendering engine 170 can be implemented in the audio-only interface device 120. In such an implementation, the query engineering engine 140 and / or modality projection engine 160 can be coupled with the network interface 175. Such an implementation relies on a larger amount of processing resources being available at the audio-only interface device 120, but can appreciably reduce use of network resources by only transmitting and / or receiving text-based information.

[0023] In operation, the speech-to-text transcription engine 125 receives the audio query input signal 105 and converts the audio query input signal 105 into a text-based input signal 135. The speech-to-text conversion can be implemented in any suitable manner. In some implementations, the audio query input signal 105 is filtered and / or otherwise preprocessed to emphasize relevant audio features while de-emphasizing other features. The signal can then be processed based on a pre-trained acoustic model (e.g., based on sequences of phonetic units and / or other speech-relevant acoustic features) and on a pre-trained language model. The pre-trained language model can be a LLM tailored for speech-to-text conversion. Implementations can then align features derived from the acoustic model and the language model to match acoustic and linguistic units to facilitate decoding using statistical models, deep-learning models, or other techniques. The decoded audio can be further processed into sequences of words, or the like, which can then be transcribed into text. Some implementations of the speech-to-text transcription engine 125 seek to output the textbased input signal 135 as a literal textual representation of the audio spoken by the consumer 110.

[0024] In some embodiments, the query engineering engine 140 generates a text-based query prompt 145 from the text-based input signal 135. The text-based query prompt 145 can be generated in any suitable manner to cause the multimodal qLLM 150 to search for relevant multimodal query results 155 based on the text-based input signal 135, and to cause the multimodal qLLM 150 to output the multimodal query results 155 along with query context data 157. The multimodal qLLM 150 is an LLM trained to communicate query information 152 with the Internet and / or any suitable databases, content sources, etc. to obtain relevant query results in any of multiple supported modalities. As noted above, it is assumed herein that the multimodal qLLM 150 supports outputting multimodal query results 155 in a set of modalities including at least one text-based modality and at least one non-text non-audio modality, such as an image, video, etc. Indeed, a specific query in a specific instance may not return results in all supported modalities. The query context data 157 generated by the multimodal qLLM 150 can include anysuitable data to describe and / or summarize the context of the query. For example, transformer features of the multimodal qLLM 150 can be used to derive a general subject matter context for the query, to generate a list of relevant keywords, or otherwise to provide a framework by which to contextualize the results.

[0025] Embodiments of the modality projection engine 160 convert each of the multimodal query results 155 to text based on the specific modality associated with the result. The modality projection engine 160 converts the multimodal query results 155 to text-based query results, such that each text-based query result corresponds to one of the multimodal query results 155. For example, a text-based multimodal query result 155 can be passed through as a text-based query result; an image-based multimodal query result 155 can be converted to a text-based query result that is effectively a relevant description of the image; etc. The modality projection engine 160 then generates a text-based query output 165 by summarizing the text-based query results based on the query context data 157. For example, the various text-based query results can be concatenated and transformed, based on the query context data 157, into a contextually appropriate response to the query.

[0026] Embodiments of the text-to-speech rendering engine 170 synthesize an audio query output from the text-based query output 165. The text-to-speech conversion can be performed in any suitable manner. In some implementations, the text-based query output 165 is analyzed based on a language model to derive linguistic properties relevant to pronunciation, such as phonemes and punctuation. Some implementations further derive prosody -related features, such as rhythm, stress, intonation, and the like. Acoustic modeling is applied to the derived linguistic, prosodic, and / or other relevant features to generate audio corresponding to relevant speech sounds. An acoustic waveform (i.e., the audio query output 107, or the received audio query output 107’) can then be generated from the generated audio data, such as by using concatenative and / or parametric synthesis techniques. In some implementations, the audio query output 107 (or received audio query output 107’) has been further processed, such as to optimize the audio for transmission, for playback, etc.

[0027] FIG. 2 shows a block diagram of an illustrative embodiment of a partial back-end system 200 for multimodal query processing, according to embodiments described herein. The partial back-end system 200 can be an implementation of corresponding portions of the back-end system 130 of FIG. 1. As illustrated, the partial back-end system 200 includes a query engineering engine 140, a multimodal qLLM 150, and a modality projection engine 160. As described with reference to FIG. 1, the query engineering engine 140 receives a text-based input signal 135 and generates atext-based query prompt 145 for the multimodal qLLM 150. The multimodal qLLM 150 uses the text-based query prompt 145 to obtain multimodal query results 155 and to generate query context data 157. For example, embodiments of the query engineering engine 140 can formulate the textbased query prompt 145 in a manner that explicitly requests generation of the query context data 157 by the multimodal qLLM 150. The multimodal query results 155 and the query context data 157 are used by the modality projection engine 160 to generate a text-based query output 165.

[0028] In the illustrated embodiment, the modality projection engine 160 includes a modality classifier 210, several modality-to-text converters 220, and a context-aware summarizer 230. Each modality -to-text converter 220 includes an LLM specially trained on a specific modality to convert from that modality into a text-based modality. For example, one of the modality-to-text converters 220 can be an image-to-text converter that is trained on a large corpus of images and corresponding image descriptions to be able to automatically generate an appropriate textual description of the contents of the image. In some embodiments, some or all of the modality-to-text converters 220 perform their respective conversions further based on the query context data 157. For example, a first query for the colors of a company logo returns an image as one of the multimodal query results 155 that shows the company logo overlaid on a chart of the company’s stock performance over the past month; and a second query for company’s stock performance over the past month returns the same image as one of the multimodal query results 155. In embodiments that perform modality-to-text conversion based on query context data 157, the conversion in context of the first query may focus on describing the colors of the logo as represented in the image, while the conversion in context of the second query may focus on describing the stock performance reflected in the chart as represented in the image.

[0029] The modality classifier 210 analyzes each multimodal query result 155 to determine a corresponding modality and determines whether and how to route the multimodal query result 155 to the modality-to-text converters 220 based on the determined corresponding modality. In some embodiments, the modality classifier 210 is configured to classify (i.e., to recognize) N modalities, where N is an integer greater than 1. The modalities can include at least one text-based modality and at least one non-text-based modality (e.g., a non-text non-audio modality, such as an imagebased or video-based modality).

[0030] The modality classifier 210 can classify the multimodal query results 155 into respective modalities in various ways. In some embodiments, the modality classifier 210 classifies the multimodal query results 155 by analyzing the result files, such as by determining file types (e.g., file extensions, etc.) associated with the multimodal query results 155 and matching known filetypes to corresponding modalities. In other embodiments, the modality classifier 210 classifies the multimodal query results 155 by analyzing metadata, such as by parsing headers and / or other information to derive modality-relevant information about the multimodal query results 155. In other embodiments, the modality classifier 210 classifies the multimodal query results 155 by analyzing a source of each multimodal query result 155, such as where certain content sources locations are known to host particular modalities (e.g., an image repository).

[0031] Turning briefly to FIG. 3, another approach is shown for classifying the multimodal query results 155 by the modality classifier 210 in a partial back-end system 300. In such an approach, the query engineering engine 140 engineers the text-based query prompt 145 to request an explicit output of classification data 305 as a result of processing by the multimodal qLLM 150. The classification data 305 is generated to indicate a modality classification for each of the plurality of multimodal query results 155. Alternatively, the multimodal qLLM 150 is configured automatically to output the classification data 305 without need for explicit instruction by the textbased query prompt 145. The modality classifier 210 can then classify each of the multimodal query results 155 based on the classification data 305 generated by the multimodal qLLM 150.

[0032] Turning back to FIG. 2, the illustrated embodiment has N-l modality -to-text converters 220 (labeled in FIG. 2 as “M2-to-Text Converter 220b” . . . “MN-to-Text Converter 220n”), one for each modality other than the text-based modality. As illustrated, the modality classifier 210 can have N classifier output paths 215 (labeled 215a - 215n), including a pass-through output path for text-based ones of the multimodal query results 155. Each of classifer output paths 215b - 215n can be coupled with a respective one of modality -to-text converters 220b - 220n, which can output converted query results via converter output paths 225b - 225n. In such an embodiment, a multimodal query result 155 determined already to be in a text-based modality can effectively be passed through by the modality classifier 210 without conversion by any of the modality -to-text converters 220, such that the corresponding classifier output path 215a is effectively also a corresponding converter output path 225a. Some embodiments can also pass some or all textbased multimodal query results 155 throuhgh a modality -to-text converter 220, such that there are N modality-to-text converters 220. For example, such a text-to-text converter can ensure that all the text output to the converter output paths 225a - 225n is in natural language (e.g., is not actually computer code, or some other use of text characters), is in a same language, is free of features that may frustrate use by the context-aware summarizer 230 (e.g., special characters), etc.

[0033] In some embodiments, some or all of the modality-to-text converters 220 perform a hierarchical sub-classification into sub-modalities. Turning briefly to FIG. 4, a partial modalityprojection engine 400 is shown, including a modality classifier 210 and an illustrative modality -to- text converter 220. The partial modality projection engine 400 can be an implementation of the modality projection engine 160 of FIG. 1 or 2. The illustrated modality -to-text converter 220 can represent a kth one of N or N-l modality-to-text converters 220 tailored to a kth one of N modalities, such as described with reference to FIG. 2. It is assumed in context of FIG. 4 that the kth modality can be further classified into up to X sub-modalities, where X is at least two. For example, an “image” modality can be further classified into sub-modalities, such as “photo,” “line drawing,” “portrait,” “financial chart,” “data plot,” etc.

[0034] As illustrated, the kth modality-to-text converter 220k (labeled as “MK-to-Text Converter 220k”) can include a sub-classifier 410 that is an artificial intelligence (e.g., machine learning) classification model specifically trained for sub-classification of the kth modality (labeled as “MK Sub-Classifier 410k”). For example, in the image modality, the sub-classifier 410 is specifically trained on one or more corpora of different types of images to build a classification model for automatically differentiating between different sub-modalities of images. In some embodiments, each of the modality-to-text converters 220 has its own sub-classifier 410. In other embodiments, only a subset (e.g., one or more, but fewer than all) of the modality-to-text converters 220 supports further sub-classification. As illustrated, the sub-classifier 410 is coupled with up to X sub-modality-to-text converters 420 (labeled as “MKl-to-Text Converter 420ka” - “MKX-to-Text Converter 420kx”). Though not explicitly shown, one or more of the modality-to- text converters 220 can include more than one level of sub-classification. For example, MK subclassifier 410k can output to one or more levels of MK sub-sub-classifiers (not shown), which can ultimately output more finely classified sub-modalities to corresponding sub-modality-to-text converters 420.

[0035] It can be seen that embodiments facilitate a modular approach to both the modality classification and the modality-to-text conversion. For example, modules can be added into the architecture of the modality projection engine 160 where desired to support finer classification distinctions, which can yield better modality to text conversion. Conversely, modules can be removed (or not added) into the architecture of the modality projection engine 160 where it is desirable to reduce resource usage (e.g., cloud memory and / or processing resources used by the back-end system).

[0036] Returning again to FIG. 2, the N converter output paths 225 can be coupled with the context-aware summarizer 230. Embodiments of the context-aware summarizer 230 include an LLM-based transformer. After conversion by the modality-to-text converters 220, all theinformation fed to the context-aware summarizer 230 is in a common text-based modality. In some cases, all the text fed to the context-aware summarizer 230 is aggregated and transformed by the context-aware summarizer 230 into a context-appropriate summary based on the query context data 157. In some implementations, the aggregation of the text includes concatenation of all the text from all the multimodal query results 155. In other implementations, the aggregation can include additional text processing, such as filtering, de-duplicating, prioritizing, etc. The query context data 157 is used to help ensure that the summary is generated in a manner that is appropriately responsive to the initial query. In some implementations, the context-aware summarizer 230 includes a prompt generator as a front-end interface to the LLM-based transformer. The prompt generator can generate a prompt that effectively asks the LLM-based transformer to answer the prompt based on the aggregated text from the multimodal query results 155.

[0037] FIG. 5 shows a flow diagram of an illustrative method 500 for multimodal query processing in context of an audio-only interface device, according to embodiments described herein. Embodiments of the method 500 can be implemented using any suitable system, including system embodiments described herein. Embodiments begin at stage 504 by converting an audio query input signal to a text-based query prompt. As described herein, some embodiments perform the converting at stage 504 by first converting the audio query input signal to a text-based input signal, and then converting the text-based input signal to the text-based query prompt.

[0038] At stage 508, embodiments can process the text-based query prompt, by a multimodal query large language model (LLM), to generate multimodal query results and query context data. As described herein, each multimodal query result is associated with one several modalities, such as a text-based modality, an image-based modality, a video-based modality, an audio-based modality, etc. As described herein, some embodiments of the multimodal query LLM can generate additional data for use by downstream components, such as classification data indicating an associated modality for each of the generated multimodal query results. In some implementations, the text-based query prompt can be generated in stage 504 to explicitly direct the multimodal query LLM to output the context data and / or the classification data. In other implementations, the multimodal query LLM is trained and configured to automatically output the context data and / or the classification data as part of outputting multimodal results.

[0039] At stage 512, embodiments can convert the multimodal query results to text-based query results, such that each text-based query result corresponds to one of the multimodal query results. As described herein, such conversion can involve classifying the multimodal query result todetermine the associated modality for each result. Based on the classifying, each multimodal query result can be routed to a selected one of multiple modality-to-text converters corresponding to the classified associated modality. Each of the modality-to-text converters can be an LLM specifically trained and tailored to convert a specific modality to text. Each of the modality-to-text converters can then generate a corresponding one of the text-based query results to describe the multimodal query result.

[0040] In some embodiments, the classifying involves determining whether the multimodal query result is associated with a text-based modality. If not, the multimodal query result can be routed to an appropriate modality-to-text converter, which can generate the corresponding one of the text-based query results. However, if the multimodal query result is determined already to be in a text-based modality, some embodiments can pass through the result without modal conversion. Alternatively, some embodiments can still convert (or validate, process, etc.) a multimodal query result determined already to be in a text-based modality, for example, to ensure that the result is a natural language result, is in a desired format, is in a desired language, is free of characters or artifacts that may frustrate downstream uses, etc.

[0041] As described herein, classification of the multimodal query results can be performed in various ways. In some implementations, the converting the audio query input signal to the textbased query prompt at stage 504 involves engineering the text-based query prompt to request an explicit output of classification data as a result of the processing in stage 508, and the classifying as part of stage 512 is based on the classification data. In other implementations, the classifying involves determining the associated one of the plurality of modalities based on analyzing a file type, metadata, a content source, and / or other information about the multimodal query result.

[0042] As described herein, generating the corresponding text-based query results as part of stage 512 can include processing each multimodal query result to identify a sub-modality associated with the multimodal query result, wherein the sub-modality is one of several submodalities of the modality determined to be associated with the multimodal query result by the classification. In such cases, the multimodal query result can be routed to a selected one of several sub-modality-to-text converters corresponding to the identified sub-modality. The selected one of the plurality of sub-modality-to-text converters can then generate the corresponding one of the text-based query results to describe the multimodal query result.

[0043] At stage 516, embodiments can generate a text-based query output by concatenating and summarizing the text-based query results based on the query context data. For example, a summary prompt can be generated for an LLM-based transformer based on the query context data,and the text-based query results can be processed by the LLM-based transformer based on the summary prompt to generate the text-based query output. In some implementations, the concatenation involves only aggregation of the text-based query results. In other implementations, the concatenation can additionally involve pre-processing of results, such as by de-duplicating, filtering, weighting, etc.

[0044] At stage 520, embodiments can synthesize an audio query output from the text-based query output. Embodiments of the method 500 can be implemented locally by a smart appliance, by a computational environment in local communication (e.g., by a wired connection, by a wireless local area network, etc.) with the smart appliance, in the cloud, or in any other suitable location. For example, embodiments can perform any or all of stages 504 - 520 in a computational environment integrated in the smart appliance (e.g., by one or more processors of the smart appliance). Alternatively, embodiments can perform some or all stages of the method 500 remotely. For example, as illustrated, some embodiments of the method 500 begin at stage 502 by receiving the audio query input via a network from a smart appliance (e.g., by a cloudbased server). For example, the audio query input is generated by the smart appliance based on a spoken query received from a user via an audio-only user interface. In some such embodiments, the method 500 can further include, at stage 522, transmitting the audio query output (generated in stage 520) via the network to the smart appliance for output by the smart appliance to the user via the audio-only user interface.

[0045] Embodiments of methods and systems described above can be implemented in a computational environment. FIG. 6 shows an illustrative distributed computational environment 600 for implementing multimodal query processing, according to embodiments described herein. The illustrated distributed computational environment 600 can be implemented in the cloud and can include cloud-based processing resources 610 and cloud-based memory resources 620. The cloud-based processing resources 610 includes one or more processors of the distributed computational environment. For example, the one or more processors include central processing units (CPUs), or other processors, of a network of computer servers, or the like. The clous-based memory resources 620 can include any suitable non-transitory, processor-readable memory that is accessible to the cloud-based processing resources 610.

[0046] Embodiments of the cloud-based memory resources 620 have instructions stored thereon. Execution of the instructions causes one or more processors of the cloud-based processing resources 610 to perform steps of the method 500 described with reference to FIG. 5, and / or to implement some or all system components described with reference to FIGS. 1 - 4. For example,as illustrated, in some embodiments, the instructions cause one or more processors of the cloudbased processing resources 610 to implement the multimodal qLLM 150 and the modality projection engine 160. In some such embodiments, the speech-to-text transcription engine 125, query engineering engine 140, and / or text-to-speech rendering engine 170 can be implemented at the end-user device. In other embodiments, the speech-to-text transcription engine 125, query engineering engine 140, and / or text-to-speech rendering engine 170 can be implemented by the cloud-based processing resources 610 responsive to execution of corresponding instructions stored by the cloud-based memory resources 620 . In addition to storing the instructions, the cloud-based memory resources 620 can be used to store the massive language models and other models underlying the multimodal qLLM 150, LLMs of the modality projection engine 160, and other components described herein.

[0047] Having described several example configurations, various modifications, alternative constructions, and equivalents may be used without departing from the spirit of the disclosure. For example, the above elements may be components of a larger system, wherein other rules may take precedence over or otherwise modify the application of the invention. Also, a number of steps may be undertaken before, during, or after the above elements are considered.

Claims

WHAT IS CLAIMED IS:

1. A method for multimodal query processing comprising: converting an audio query input signal to a text-based query prompt; processing the text-based query prompt by a multimodal query large language model (LLM) to generate a plurality of multimodal query results and query context data, each query result associated with one of a plurality of modalities; converting the plurality of multimodal query results to a plurality of text-based query results, such that each text-based query result corresponds to one of the plurality of multimodal query results; generating a text-based query output by concatenating and summarizing the plurality of text-based query results based on the query context data; and synthesizing an audio query output from the text-based query output.

2. The method of claim 1, wherein the converting the audio query input signal to the text-based query prompt comprises engineering the text-based query prompt to request an explicit output of the query context data as a result of the processing.

3. The method of claim 1, wherein the converting the plurality of multimodal query results to the plurality of text-based query results comprises, for each of the plurality of multimodal query results: classifying the multimodal query result to determine the associated one of the plurality of modalities; routing the multimodal query result to a selected one of a plurality of modality -to- text converters corresponding to the associated one of the plurality of modalities based on the classifying; and generating, by the selected one of the plurality of modality-to-text converters, a corresponding one of the text-based query results to describe the multimodal query result.

4. The method of claim 3, wherein: the classifying comprises determining whether the multimodal query result is associated with a text-based modality; the routing and the generating the corresponding one of the text-based query results are performed if the multimodal query result is determined not to be associated with the text-based modality; andthe converting the plurality of multimodal query results to the plurality of textbased query results comprises using the multimodal query result as the corresponding text-based query result if the multimodal query result is determined to be associated with the text-based modality.

5. The method of claim 3, wherein: the converting the audio query input signal to the text-based query prompt comprises engineering the text-based query prompt to request an explicit output of classification data as a result of the processing, the classification data indicating a modality classification for each of the plurality of multimodal query results; and the classifying is based on the classification data.

6. The method of claim 3, wherein: the classifying comprises determining the associated one of the plurality of modalities based on analyzing a file type and / or metadata of the multimodal query result.

7. The method of claim 3, wherein the generating the corresponding one of the text-based query results comprises: processing the multimodal query result to identify a sub-modality associated with the multimodal query result, the sub-modality being one of a plurality of sub-modalities of the associated one of the plurality of modalities; routing the multimodal query result to a selected one of a plurality of sub-modality - to-text converters corresponding to the identified sub-modality; and generating, by the selected one of the plurality of sub-modality-to-text converters, the corresponding one of the text-based query results to describe the multimodal query result.

8. The method of claim 1, wherein the generating the text-based query output comprises: generating a summary prompt for an LLM-based transformer based on the query context data; and processing the plurality of text-based query results by the LLM-based transformer based on the summary prompt to generate the text-based query output.

9. The method of claim 1, further comprising: receiving the audio query input via a network from a smart appliance, the audio query input generated by the smart appliance based on a spoken query received from a user via an audio-only user interface; andtransmitting the audio query output via the network to the smart appliance for output by the smart appliance to the user via the audio-only user interface.

10. A system comprising: one or more processors; and one or more non-transitory, processor-readable memories having instructions stored thereon which, when executed, cause the one or more processors to perform the method of claim 1.

11. A system for multimodal query processing comprising: a speech-to-text transcription engine to convert an audio query input signal to a text-based input signal; a query engineering engine to generate a text-based query prompt from the textbased input signal; a multimodal query large language model (LLM) to process the text-based query prompt to generate a plurality of multimodal query results and query context data, each multimodal query result associated with one of a plurality of modalities; a modality projection engine to: convert the plurality of multimodal query results to a plurality of text-based query results, such that each text-based query result corresponds to one of the plurality of multimodal query results; and generate a text-based query output by summarizing the plurality of textbased query results based on the query context data; and a text-to-speech rendering engine to synthesize an audio query output from the textbased query output.

12. The system of claim 11, wherein the query engineering engine is to generate the text-based query prompt by engineering the text-based query prompt to request an explicit output of the query context data as a result of the processing.

13. The system of claim 11, wherein the modality projection engine comprises: a plurality of modality -to-text converters, each to convert from a respective one of the plurality of modalities to a text-based modality; and a modality classifier to, for each multimodal query result: classify the multimodal query result to determine the associated one of the plurality of modalities; androute the multimodal query result to a selected one of the plurality of modality -to-text converters based on the classifying, wherein the modality projection engine is to convert the plurality of multimodal query results to the plurality of text-based query results by, for each of the plurality of multimodal query results, using the selected one of the plurality of modality-to-text converters to generate the corresponding one of the text-based query results to describe the multimodal query result.

14. The system of claim 13, wherein: each of the plurality of modality-to-text converters comprises a respective singlemodality LLM trained to convert from the respective one of the plurality of modalities to the textbased modality.

15. The system of claim 13, wherein: the modality classifier is to classify further by determining whether the multimodal query result is associated with a text-based modality; the modality classifier is to route the multimodal query result to a selected one of the plurality of modality-to-text converters only if the multimodal query result is determined not to be associated with the text-based modality, such that the modality projection engine is to use the multimodal query result as the corresponding text-based query result without performing the converting if the multimodal query result is determined to be associated with the text-based modality.

16. The system of claim 13, wherein: the query engineering engine is to engineer the text-based query prompt to request an explicit output of classification data as a result of the processing, the classification data indicating a modality classification for each of the plurality of multimodal query results; and the modality classifier is to classify based on the classification data.

17. The system of claim 13, wherein: the modality classifier is to classify further by determining the associated one of the plurality of modalities based on analyzing a file type and / or metadata of the multimodal query result.

18. The system of claim 13, wherein: at least one of the plurality of modality-to-text converters comprises a modality sub-classifier and a plurality of sub-modality-to-text converters; andthe using the selected one of the plurality of modality -to-text converters to generate the corresponding one of the text-based query results to describe the multimodal query result comprises: processing the multimodal query result, by the modality sub-classifier, to identify a sub-modality associated with the multimodal query result, the sub-modality being one of a plurality of sub-modalities of the associated one of the plurality of modalities; routing the multimodal query result to a selected one of the plurality of submodality -to-text converters corresponding to the identified sub-modality; and generating, by the selected one of the plurality of sub-modality-to-text converters, the corresponding one of the text-based query results to describe the multimodal query result.

19. The system of claim 11, wherein the modality projection engine comprises: a context-aware summarizer having an LLM-based transformer and configured to generate the text-based query output by: generating a summary prompt for the LLM-based transformer based on the query context data; and processing the plurality of text-based query results by the LLM-based transformer based on the summary prompt to generate the text-based query output.

20. The system of claim 11, further comprising: a network interface to communicatively couple, via one or more communication networks, with a smart appliance having an audio-only user interface, wherein the speech-to-text transcription engine is coupled with the network interface to receive the audio query input from the smart appliance via the one or more communication networks, the audio query input generated by the smart appliance based on a spoken query received from a user via the audio-only user interface, and wherein the text-to-speech rendering engine is coupled with the network interface to transmit the audio query output to the smart appliance via the one of more communication networks for output by the smart appliance to the user via the audio-only user interface.

Images