Zero-sample multi-modal data processing via structured inter-model communication

Abstract

Systems and methods of the present disclosure relate to a computer-implemented method for context processing via inter-model communication between pre-trained machine learning models. The method includes obtaining, by a computing system including one or more computing devices, input data. The method includes processing, by a computing system, input data with two or more pre-trained models to generate output data, wherein processing the input includes executing a structured inter-model communication mode to perform inter-model communication between the two or more pre-trained models over a communication channel. The method includes providing, by the computing system, the output data as an output.

Description

[0001] Priority claim

[0002] This application is based upon and claims priority from U.S. Provisional Application No. 63 / 326,643, filed April 1, 2022, which is incorporated herein by reference. Technical Field

[0003] The present disclosure generally relates to structured inter-model communication for machine learning models. More specifically, the present disclosure relates to contextual processing of multimodal data via structured inter-model communication between underlying machine learning models. Background Art

[0004] Base models are models trained on a wide range of data at scale and are applicable to a wide variety of downstream tasks (e.g., visual language models (VLMs), large language models (LMs), audio language models (ALMs), etc.). Recently, base models have achieved impressive capabilities for a variety of machine learning tasks. However, these capabilities depend on the distribution of the training data, which often varies significantly across domains. For example, VLMs are typically trained on images and video captions, while LMs are trained on large corpora of other data (e.g., spreadsheets, fictional novels, standardized test questions, etc.). Summary of the Invention

[0005] Aspects and advantages of embodiments of the present disclosure will be set forth in part in the following description, or may be learned from the description, or may be learned through practice of the embodiments.

[0006] One example aspect of the present disclosure relates to a computer-implemented method for context processing via inter-model communication between machine learning models. The method includes obtaining input data by a computing system comprising one or more computing devices. The method includes processing the input data using two or more pre-trained models by the computing system to generate output data, wherein processing the input includes executing a structured inter-model communication pattern between the two or more pre-trained models. The method includes providing the output data as output by the computing system.

[0007] Another example aspect of the present disclosure relates to a computing system for performing context processing using a base machine learning model. The computing system includes one or more processors. The computing system includes one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations. The operations include obtaining input data. The operations include processing the input data using two or more pre-trained models to generate output data, wherein processing the input includes executing a structured inter-model communication pattern between the two or more pre-trained models. The operations include providing the output data as output.

[0008] Another example aspect of the present disclosure relates to one or more non-transitory computer-readable media that collectively store instructions that, when executed by one or more computing devices, cause the one or more computing devices to perform operations. The operations include obtaining input data. The operations include processing the input data using two or more pre-trained models to generate output data, wherein processing the input includes executing a structured inter-model communication mode (structured dialogue) between the two or more pre-trained models. The operations include providing the output data as output.

[0009] Another example aspect of the present disclosure relates to a method for performing context processing via structured inter-model communication between machine learning models. The method includes obtaining, by a computing system comprising one or more computing devices, input data and a context data corpus, wherein the input data comprises data describing a query, and wherein the context data corpus comprises multimodal data. The method includes processing, by the computing system, the context data corpus using one or more of two or more pre-trained models to obtain a language-based context history, wherein the one or more pre-trained models comprise a language model.

[0010] Other aspects of the disclosure relate to various systems, apparatuses, non-transitory computer-readable media, user interfaces, and electronic devices.

[0011] These and other features, aspects and advantages of various embodiments of the present disclosure will be better understood with reference to the following description and appended claims.The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate example embodiments of the present disclosure and, together with the description, serve to explain the relevant principles. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] With reference to the accompanying drawings, a detailed discussion of the embodiments for persons of ordinary skill in the art is set forth in this specification, in which:

[0013] Figure 1ADepicted is a block diagram of an example computing system that performs context processing via structured inter-model communication between pre-trained machine learning models, according to an example embodiment of the present disclosure.

[0014] Figure 1B Depicted is a block diagram of an example computing device that performs context processing via structured inter-model communication between pre-trained machine learning models, according to an example embodiment of the present disclosure.

[0015] Figure 1C Depicted is a block diagram of an example computing device that performs context processing via structured inter-model communication between pre-trained machine learning models, according to an example embodiment of the present disclosure.

[0016] Figure 2 Depicted is a block diagram of an example pre-trained model and associated structured inter-model communication pattern, according to an example embodiment of the present disclosure.

[0017] Reference numerals that are repeated across multiple figures are intended to identify like features in various implementations. DETAILED DESCRIPTION

[0018] Overview

[0019] In general, the present disclosure relates to contextual zero-shot processing via structured inter-model communication between machine learning models (e.g., base models). More specifically, recent advances in machine learning have enabled the creation of large models (i.e., "base models") that are capable of performing a wide variety of downstream zero-shot tasks. The present disclosure uses a structured (i.e., Socratic) inter-model communication pattern to leverage the complementary differences between existing pre-trained base models to perform new tasks without any additional training or fine-tuning.

[0020] Specifically, structured inter-model communication can be used to guide exchanges between base models and thus exploit their zero-shot capabilities. The structured inter-model communication pattern can be executed to process input data with two or more pre-trained models and generate output data (e.g., video data, text data, etc.). As an example, the structured inter-model communication pattern can instruct a pre-trained language model to process a query input from a user (e.g., "where is the remote control") to obtain a prompt (e.g., "remote control"). The structured inter-model communication pattern can then instruct a pre-trained visual language model to process the prompt to obtain a series of keyframes (e.g., frames depicting the last known position of the remote control) from first-person video data collected by the user. In this way, the complementary zero-shot capabilities of multiple base models can be exploited to perform new and increasingly complex tasks without any additional training or fine-tuning.

[0021] Embodiments of the present disclosure provide a variety of technical effects and benefits. As an example technical effect and benefit, by leveraging the complementary capabilities of base models via structured inter-model communication, embodiments of the present disclosure provide significant improvements in various machine learning use cases compared to conventional techniques (e.g., question / answering tasks, video understanding tasks, prediction tasks, etc.). As another example technical effect and benefit, the base model is a very large model that requires a lot of resources to train, retrain, or otherwise optimize. By avoiding the need to retrain or fine-tune any existing pre-trained model, embodiments of the present disclosure eliminate the large amount of computing resources (e.g., computing cycles, memory, power, storage devices, hardware, etc.) associated with retraining that would be required using conventional techniques.

[0022] Referring now to the accompanying drawings, example embodiments of the present disclosure will be discussed in greater detail.

[0023] Example devices and systems

[0024] Figure 1A A block diagram of an example computing system for performing context processing via structured inter-model communication between pre-trained machine learning models is depicted, according to an example embodiment of the present disclosure. System 100 includes a user computing device 102, a server computing system 130, and a training computing system 150 communicatively coupled via a network 180.

[0025] The user computing device 102 can be any type of computing device, such as, for example, a personal computing device (e.g., a laptop or desktop computer), a mobile computing device (e.g., a smartphone or tablet computer), a gaming console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.

[0026] The user computing device 102 includes one or more processors 112 and memory 114. The one or more processors 112 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and can be a single processor or multiple processors operatively connected. The memory 114 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 114 can store data 116 and instructions 118 that are executed by the processor 112 to cause the user computing device 102 to perform operations.

[0027] In some implementations, the user computing device 102 may store or include one or more pre-trained base models 120 (e.g., a large language model, a visual language model, an audio language model, etc.). For example, the pre-trained base model 120 may be or may otherwise include various machine learning models, such as a neural network (e.g., a deep neural network) or other types of machine learning models, including non-linear models and / or linear models. The neural network may include a feedforward neural network, a recurrent neural network (e.g., a long short-term memory recurrent neural network), a convolutional neural network, or other forms of neural networks. Some example machine learning models may utilize attention mechanisms, such as self-attention. For example, some example machine learning models may include a multi-head self-attention model (e.g., a Transformer model). An example pre-trained base model 120 is discussed with reference to FIG4. It should be noted that although the present disclosure is described with respect to a base model, the base model is not necessary for utilization of the present disclosure. Instead, in some embodiments, a non-base model may replace the base model.

[0028] In some implementations, one or more pre-trained base models 120 may be received from the server computing system 130 via the network 180, stored in the user computing device memory 114, and then used or otherwise implemented by the one or more processors 112. In some implementations, the user computing device 102 may implement multiple parallel instances of a single pre-trained base model 120 (e.g., to perform parallel context processing across multiple instances of the base model 120).

[0029] More specifically, the pre-trained base machine learning model 120 can be utilized via structured inter-model communication (e.g., using a pattern and communication over a communication channel) to perform contextual processing (e.g., zero-shot processing) that enables the execution of previously untrained tasks without requiring any additional training of the model. For example, the user computing device 102 can obtain input data that includes data indicating a query (e.g., via the user input component 122) and multimodal data (e.g., video data, audio data, text data, etc.) (e.g., via sensors of the device 102, etc.). The user computing device 102 can execute a structured inter-model communication pattern to perform inter-model communication between two pre-trained models 120 via a communication channel to process the input data, thereby generating output data. The output data can be provided as output by the user computing device 102 (e.g., provided to a user of the user computing device 102, etc.).

[0030] As an example, the pre-trained base machine learning model 120 may include a language model and a visual language model. To implement the structured inter-model communication pattern, the user computing device 102 may process data describing a query (e.g., text input data, etc.) using a model from the pre-trained model 102 (e.g., a language model, etc.) to obtain a prompt associated with the query. The user computing device 102 may process the prompt using the visual language models of two or more pre-trained models 102 to obtain output data including one or more video frames associated with the prompt. For a specific example, the query may ask "When was the last time I saw my remote control?" and the multimodal data may include video data recorded from the user computing device 102 or transmitted to the user device, which captures a first-person view from the user over a period of time. The language model may process the query to obtain a prompt including "remote control." The visual language model may be used to process the "remote control" prompt to obtain output data that may include one or more video frames depicting the remote control when the remote control was last seen. In this manner, structured inter-model communication may occur over the communication channel to provide contextual processing, thereby providing the user computing device 102 with the ability to perform additional tasks without utilizing additional resources to further train the pre-trained model 120 .

[0031] Additionally or alternatively, one or more pre-trained base models 140 may be included in or otherwise stored and implemented by a server computing system 130 that communicates with the user computing device 102 according to a client-server relationship. For example, the pre-trained base models 140 may be implemented by the server computing system 140 as part of a web service (e.g., a context processing (e.g., zero-shot processing) service). Thus, one or more models 120 may be stored and implemented at the user computing device 102, and / or one or more models 140 may be stored and implemented at the server computing system 130.

[0032] The user computing device 102 may also include one or more user input components 122 for receiving user input. For example, the user input component 122 may be a touch-sensitive component (e.g., a touch-sensitive display or touchpad) that is sensitive to the touch of a user input object (e.g., a finger or a stylus). A touch-sensitive component may be used to implement a virtual keyboard. Other example user input components include a microphone, a traditional keyboard, or other devices by which a user can provide user input.

[0033] The server computing system 130 includes one or more processors 132 and memory 134. The one or more processors 132 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and can be a single processor or multiple processors operatively connected. The memory 134 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 134 can store data 136 and instructions 138 that are executed by the processor 132 to cause the server computing system 130 to perform operations.

[0034] In some implementations, the server computing system 130 includes or is otherwise implemented by one or more server computing devices. Where the server computing system 130 includes multiple server computing devices, such server computing devices may operate according to a sequential computing architecture, a parallel computing architecture, or some combination thereof.

[0035] As described above, the server computing system 130 may store or otherwise include one or more pre-trained base models 140. For example, the model 140 may be or may otherwise include various machine learning models. Example machine learning models include neural networks or other multi-layer nonlinear models. Example neural networks include feedforward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks. Some example machine learning models may utilize attention mechanisms, such as self-attention. For example, some example machine learning models may include a multi-head self-attention model (e.g., a Transformer model). Example models 140 are discussed with reference to FIG4.

[0036] Optionally, user computing device 102 and / or server computing system 130 may train models 120 and / or 140 via interaction with a training computing system 150 communicatively coupled via network 180. Training computing system 150 may be separate from server computing system 130 or may be part of server computing system 130.

[0037] Optionally, the training computing system 150 includes one or more processors 152 and memory 154. The one or more processors 152 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and can be a single processor or multiple processors operatively connected. The memory 154 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 154 can store data 156 and instructions 158 that are executed by the processor 152 to cause the training computing system 150 to perform operations. In some implementations, the training computing system 150 includes or is otherwise implemented by one or more server computing devices.

[0038] The training computing system 150 may include a model trainer 160 that trains the machine learning models 120 and / or 140 stored at the user computing device 102 and / or the server computing system 130 using various training or learning techniques (e.g., such as error backpropagation). For example, a loss function may be backpropagated through the model to update one or more parameters of the model (e.g., based on the gradient of the loss function). Various loss functions may be used, such as mean squared error, likelihood loss, cross entropy loss, hinge loss, and / or various other loss functions. Gradient descent techniques may be used to iteratively update the parameters over multiple training iterations.

[0039] In some implementations, performing error backpropagation may include performing truncated backpropagation through time.The model trainer 160 may perform various generalization techniques (eg, weight decay, dropout, etc.) to improve the generalization ability of the model being trained.

[0040] In some implementations, if the user has provided consent, training examples may be provided by the user computing device 102. Thus, in such implementations, the model 120 provided to the user computing device 102 may be trained by the training computing system 150 based on user-specific data received from the user computing device 102. In some cases, this process may be referred to as personalizing the model.

[0041] The model trainer 160 includes computer logic for providing the desired functionality. The model trainer 160 can be implemented using hardware, firmware, and / or software that controls a general-purpose processor. For example, in some implementations, the model trainer 160 includes a program file stored on a storage device, loaded into a memory, and executed by one or more processors. In other implementations, the model trainer 160 includes one or more sets of computer-executable instructions stored in a tangible computer-readable storage medium, such as RAM, a hard disk, or optical or magnetic media.

[0042] However, it should be noted that embodiments of the present disclosure provide the pre-trained model 120 with the ability to perform previously untrained tasks without requiring additional training via a training computing system or any other system. As such, some embodiments of the present disclosure may avoid the need to utilize a training computing system 150 and any other training systems or techniques.

[0043] The network 180 can be any type of communication network, such as a local area network (e.g., an intranet), a wide area network (e.g., the Internet), or some combination thereof, and can include any number of wired or wireless links. Generally, communications over the network 180 can be conducted over any type of wired and / or wireless connection using a variety of communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and / or protection schemes (e.g., VPN, secure HTTP, SSL).

[0044] The machine learning models described in this specification can be used for a variety of tasks, applications, and / or use cases.

[0045] In some implementations, the input to the machine learning model of the present disclosure may be image data. The machine learning model may process the image data to generate an output. As an example, the machine learning model may process the image data to generate an image recognition output (e.g., recognition of the image data, potential embedding of the image data, an encoded representation of the image data, a hash of the image data, etc.). As another example, the machine learning model may process the image data to generate an image segmentation output. As another example, the machine learning model may process the image data to generate an image classification output. As another example, the machine learning model may process the image data to generate an image data modification output (e.g., a change to the image data, etc.). As another example, the machine learning model may process the image data to generate an encoded image data output (e.g., an encoded and / or compressed representation of the image data, etc.). As another example, the machine learning model may process the image data to generate an amplified image data output. As another example, the machine learning model may process the image data to generate a prediction output.

[0046] In some implementations, the input to the machine learning model of the present disclosure may be text or natural language data. The machine learning model may process the text or natural language data to generate an output. As an example, the machine learning model may process the natural language data to generate a language encoding output. As another example, the machine learning model may process the text or natural language data to generate a latent text embedding output. As another example, the machine learning model may process the text or natural language data to generate a translation output. As another example, the machine learning model may process the text or natural language data to generate a classification output. As another example, the machine learning model may process the text or natural language data to generate a text segmentation output. As another example, the machine learning model may process the text or natural language data to generate a semantic intent output. As another example, the machine learning model may process the text or natural language data to generate an amplified text or natural language output (e.g., text or natural language data of higher quality than the input text or natural language, etc.). As another example, the machine learning model may process the text or natural language data to generate a prediction output.

[0047] In some implementations, the input to the machine learning model of the present disclosure may be speech data. The machine learning model may process the speech data to generate an output. As an example, the machine learning model may process the speech data to generate a speech recognition output. As another example, the machine learning model may process the speech data to generate a speech translation output. As another example, the machine learning model may process the speech data to generate a potential embedding output. As another example, the machine learning model may process the speech data to generate an encoded speech output (e.g., an encoded and / or compressed representation of the speech data, etc.). As another example, the machine learning model may process the speech data to generate an amplified speech output (e.g., speech data of higher quality than the input speech data, etc.). As another example, the machine learning model may process the speech data to generate a text representation output (e.g., a text representation of the input speech data, etc.). As another example, the machine learning model may process the speech data to generate a prediction output.

[0048] In some implementations, the input to the machine learning models of the present disclosure may be statistical data. Statistical data may be, represent, or otherwise include data calculated and / or computed from some other data source. The machine learning model may process the statistical data to generate an output. As one example, the machine learning model may process the statistical data to generate an identification output. As another example, the machine learning model may process the statistical data to generate a prediction output. As another example, the machine learning model may process the statistical data to generate a classification output. As another example, the machine learning model may process the statistical data to generate a segmentation output. As another example, the machine learning model may process the statistical data to generate a visualization output. As another example, one or more machine learning models may process the statistical data to generate a diagnostic output.

[0049] In some implementations, the input to one or more machine learning models of the present disclosure may be sensor data. The machine learning model may process the sensor data to generate an output. As an example, the machine learning model may process the sensor data to generate a recognition output. As another example, the machine learning model may process the sensor data to generate a prediction output. As another example, the machine learning model may process the sensor data to generate a classification output. As another example, the machine learning model may process the sensor data to generate a segmentation output. As another example, the machine learning model may process the sensor data to generate a visualization output. As another example, the machine learning model may process the sensor data to generate a diagnostic output. As another example, the machine learning model may process the sensor data to generate a detection output.

[0050] In some cases, the input includes visual data, and the task is a computer vision task. In some cases, the input includes pixel data for one or more images, and the task is an image processing task. For example, the image processing task may be image classification, where the output is a set of scores, each score corresponding to a different object class and representing the likelihood that the one or more images depict an object belonging to that object class. The image processing task may be object detection, where the image processing output identifies one or more regions in one or more images and, for each region, identifies the likelihood that the region depicts an object of interest. As another example, the image processing task may be image segmentation, where the image processing output defines, for each pixel in one or more images, a corresponding likelihood for each of a predetermined set of categories. For example, the set of categories may be foreground and background. As another example, the set of categories may be object classes. As another example, the image processing task may be depth estimation, where the image processing output defines a corresponding depth value for each pixel in one or more images. As another example, the image processing task may be motion estimation, where the network input includes multiple images, and the image processing output defines, for each pixel in one of the input images, the motion of the scene depicted at that pixel between the images in the network input.

[0051] In some cases, the input includes audio data representing spoken utterances, and the task is a speech recognition task. The output may include a text output mapped to the spoken utterances. In some cases, the task includes encrypting or decrypting the input data. In some cases, the task includes a microprocessor performance task, such as branch prediction or memory address translation.

[0052] Figure 1A An example computing system that can be used to implement the present disclosure is shown. Other computing systems may also be used. For example, in some implementations, the user computing device 102 may include a model trainer 160 and a training dataset 162. In such implementations, the model 120 may be both trained and used locally at the user computing device 102. In some of such implementations, the user computing device 102 may implement the model trainer 160 to personalize the model 120 based on user-specific data.

[0053] Figure 1B A block diagram of an example computing device that performs context processing via structured inter-model communication between pre-trained machine learning models according to an example embodiment of the present disclosure is depicted. The computing device 10 can be a user computing device or a server computing device.

[0054] Computing device 10 includes multiple applications (e.g., Application 1 through Application N). Each application includes its own machine learning library and machine learning model. For example, each application may include a machine learning model. Example applications include text messaging applications, email applications, dictation applications, virtual keyboard applications, browser applications, etc.

[0055] like Figure 1B As shown, each application can communicate with multiple other components of the computing device (e.g., such as one or more sensors, a context manager, a device state component, and / or additional components). In some implementations, each application can communicate with each device component using an API (e.g., a public API). In some implementations, the API used by each application is specific to that application.

[0056] Figure 1C A block diagram of an example computing device that performs context processing via structured inter-model communication between pre-trained machine learning models according to an example embodiment of the present disclosure is depicted. The computing device 50 can be a user computing device or a server computing device.

[0057] The computing device 50 includes a plurality of applications (e.g., Application 1 through Application N). Each application communicates with a central intelligence layer. Example applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, and the like. In some implementations, each application can communicate with the central intelligence layer (and the models stored therein) using an API (e.g., a public API across all applications).

[0058] The central intelligence layer includes multiple machine learning models. For example, Figure 1C As shown, a corresponding machine learning model can be provided for each application, and the corresponding machine learning model can be managed by the central intelligence layer. In other implementations, two or more applications can share a single machine learning model. For example, in some implementations, the central intelligence layer can provide a single model for all applications. In some implementations, the central intelligence layer is included in the operating system of the computing device 50 or is otherwise implemented by the operating system.

[0059] The central intelligence layer can communicate with the central device data layer. The central device data layer can be a centralized data repository for the computing device 50. Figure 1C As shown, the central device data layer can communicate with multiple other components of the computing device (e.g., such as one or more sensors, context managers, device state components, and / or additional components). In some implementations, the central device data layer can communicate with each device component using an API (e.g., a private API).

[0060] Example model layout

[0061] Figure 2 A block diagram depicts an example pre-trained model and associated structured inter-model communication pattern according to an example embodiment of the present disclosure. More specifically, a computing system (e.g., computing system 130 of FIG. 1 , etc.) may execute a structured inter-model communication pattern 206 to process input data 202 using a pre-trained model 204 (e.g., pre-trained models 204A to 204C) to obtain output data 208. For example, the input data 202 may include data indicating a query (e.g., from a user, etc.) and multimodal data (e.g., video data, audio data, text data, etc.). The pre-trained machine learning model 204 may include a language model and a visual language model. To execute the structured inter-model communication pattern 206, a model (e.g., a language model, etc.) in the pre-trained model 204 may be used to process data describing the query in the input data 202 to obtain a prompt associated with the query. The prompt may be processed using the visual language model of the pre-trained model 204 to obtain output data 208 including one or more video frames associated with the prompt.

[0062] For a specific example, the query in the input data 202 may ask "When was the last time I saw my remote control?" and the multimodal data may include video data recorded from or transmitted to the user computing device 102 that captures a first-person view from the user over a period of time. A language model (e.g., pre-trained model 204A) may process the query to obtain a prompt including "remote control." A visual language model (e.g., 204B) may be used to process the "remote control" prompt to obtain output data 208, which may include one or more video frames depicting the remote control when the remote control was last seen.

[0063] The structured inter-model communication pattern 206 may be or otherwise include a series of instructions that indicate the order in which the pre-trained models 204A-204C are to process the various input data 202, the various intermediate inputs / outputs, and the output data 208. Following the previous example, the structured inter-model communication pattern 206 may instruct the pre-trained language model 204A to first process the query in the input data 202 to obtain a prompt (e.g., intermediate inputs / outputs). The structured inter-model communication pattern 206 may then instruct the pre-trained visual language model 204B to process the prompt and the multimodal data of the input data 202 to obtain output data 208 (e.g., retrieve keyframes in response to the prompt, etc.). Alternatively, in some embodiments, the structured inter-model communication pattern 206 may determine that the user desires a description rather than an image and may instruct the pre-trained visual language model to process the keyframes to obtain a description for each keyframe. Next, the structured inter-model communication pattern 206 may instruct the language model 204A to process the description of each keyframe to generate output data 208, which may include a contextual answer to the user's query.

[0064] In this way, the structured inter-model communication pattern 206 can determine the process and operation of the pre-trained model 204 based on the content of the input data 202. In addition, in some embodiments, the structured inter-model communication pattern 206 can provide structured hints to the pre-trained model 204 to further optimize the performance of the pre-trained model 204.

[0065] Additional Disclosure

[0066] The technology discussed herein relates to servers, databases, software applications, and other computer-based systems, as well as the actions taken and information sent to and from such systems. The inherent flexibility of computer-based systems allows for a variety of possible configurations, combinations, and partitioning of tasks and functions between and among components. For example, the processes discussed herein can be implemented using a single device or component or multiple devices or components working in combination. Databases and applications can be implemented on a single system or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0067] Although the present invention has been described in detail with respect to various specific example embodiments thereof, each example is provided by way of explanation, not limitation of the present invention. Those skilled in the art, upon understanding the foregoing, may readily produce modifications, changes, and equivalents to these embodiments. Therefore, the present invention does not exclude such modifications, changes, and / or additions to the present invention as would be readily understood by those of ordinary skill in the art. For example, a feature shown or described as part of one embodiment may be used in conjunction with another embodiment to produce yet another embodiment. Therefore, the present invention is intended to encompass such modifications, changes, and equivalents.

Claims

1. A computer-implemented method for context processing via structured inter-model communication between machine learning models, the method comprising: obtaining input data by a computing system including one or more computing devices; processing, by the computing system, the input data using two or more pre-trained models to generate output data, wherein processing the input includes executing a structured inter-model communication mode to perform inter-model communication between the two or more pre-trained models over a communication channel; as well as The output data is provided as output by the computing system.

2. The computer-implemented method of claim 1 , wherein the method further comprises: receiving, by the computing system, a corpus of contextual data; as well as The context data corpus is processed by the computing system with one or more of the two or more pre-trained models to obtain a language-based context history, wherein the one or more pre-trained models include a pre-trained language model.

3. The computer-implemented method of claim 2, wherein the contextual data corpus comprises multimodal data, the multimodal data comprising video data, audio data, and / or text data.

4. The computer-implemented method of claim 1 , wherein the two or more pre-trained models include two or more of: Pre-trained language models; Pre-trained vision-language models; or Pre-trained audio language models.

5. The computer-implemented method of claim 1 , wherein: The input data includes multimodal data, the multimodal data including video data and data describing a query; and Executing the structured inter-model communication mode between the two or more pre-trained models includes: Processing, by the computing system, the data describing the query using a pre-trained model of the two or more pre-trained models to obtain a hint associated with the query; as well as The prompt associated with the query is processed by the computing system using a pre-trained visual-language model of the two or more pre-trained models to obtain output data including one or more video frames associated with the prompt.

6. The computer-implemented method of claim 5, wherein: The data describing the query includes audio data or text data; and A model among the two or more pre-trained models comprises a pre-trained language model or a pre-trained audio language model.

7. The computer-implemented method of claim 4, wherein the input data comprises multimodal data, the multimodal data comprising video data; and Executing the structured inter-model communication mode between the two or more pre-trained models includes: For one or more iterations: providing, by the computing system, one or more structured cues to a pre-trained visual language model of the two or more pre-trained models to obtain data describing one or more key frames of the video data; as well as The computing system processes the data describing the one or more key frames using a pre-trained language model from among the two or more pre-trained models to obtain a natural language summary of the one or more key frames of the video data and the one or more structured prompts.

8. The computer-implemented method of claim 7, wherein: The multimodal data also includes text data describing the query; and Executing the structured inter-model communication mode between the two or more pre-trained models includes: determining, by the computing system, a language-based context history based at least in part on the one or more natural language summaries from the one or more corresponding iterations; and The language-based context history and the text data are processed by the computing system using the pre-trained language model of the two or more pre-trained models to obtain output data describing an answer to the query.

9. The computer-implemented method of claim 1, wherein the output comprises a zero-sample processing output.

10. A computing system for context processing via inter-model communication between pre-trained machine learning models, the computing system comprising: one or more processors; as well as one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations comprising: Get input data; Processing the input data with two or more pre-trained models to generate output data, wherein processing the input includes executing a structured inter-model communication mode to perform inter-model communication between the two or more pre-trained models through a communication channel; and The output data is provided as output.

11. One or more non-transitory computer-readable media collectively storing instructions that, when executed by one or more computing devices, cause the one or more computing devices to perform operations comprising: Get input data; Processing the input data with two or more pre-trained models to generate output data, wherein processing the input includes executing a structured inter-model communication mode to perform inter-model communication between the two or more pre-trained models through a communication channel; providing the output data as output; receiving a corpus of context data; The context data corpus is processed with one or more of the two or more pre-trained models to obtain a language-based context history, wherein the one or more pre-trained models include a pre-trained language model.

12. The one or more non-transitory computer-readable media of claim 11, wherein the contextual data corpus comprises multimodal data, the multimodal data comprising video data, audio data, and / or text data.

13. The one or more non-transitory computer-readable media of claim 11, wherein the two or more pre-trained models include two or more of: Pre-trained language models; Pre-trained vision-language models; or Pre-trained audio language models.

14. The one or more non-transitory computer-readable media of claim 11, wherein: The input data includes multimodal data, the multimodal data including video data and data describing a query; and Executing the structured inter-model communication mode between the two or more pre-trained models includes: processing the data describing the query with a pre-trained model of the two or more pre-trained models to obtain a hint associated with the query; as well as The prompt associated with the query is processed with a pre-trained visual-language model of the two or more pre-trained models to obtain output data including one or more video frames associated with the prompt.

15. The one or more non-transitory computer-readable media of claim 14, wherein: The data describing the query includes audio data or text data; and A model among the two or more pre-trained models comprises a pre-trained language model or a pre-trained audio language model.

16. The one or more non-transitory computer-readable media of claim 13, wherein the input data comprises multimodal data, the multimodal data comprising video data; and Executing the structured inter-model communication mode between the two or more pre-trained models includes: For one or more iterations: providing one or more structured cues to a pre-trained visual language model of the two or more pre-trained models to obtain data describing one or more key frames of the video data; as well as The data describing the one or more key frames is processed using a pre-trained language model from among the two or more pre-trained models to obtain a natural language summary of the one or more key frames of the video data and the one or more structured prompts.

17. The one or more non-transitory computer-readable media of claim 16, wherein: The multimodal data also includes text data describing the query; and Executing the structured inter-model communication mode between the two or more pre-trained models includes: determining, by the computing system, a language-based context history based at least in part on the one or more natural language summaries from the one or more corresponding iterations; and The language-based context history and the text data are processed by the computing system using the pre-trained language model of the two or more pre-trained models to obtain output data describing an answer to the query.

18. The one or more non-transitory computer-readable media of claim 11, wherein the output comprises a zero sample processing output.

19. A method for Socratic context processing via inter-model communication between pre-trained machine learning models, the method comprising: obtaining, by a computing system comprising one or more computing devices, input data and a corpus of context data, wherein the input data comprises data describing a query, and wherein the corpus of context data comprises multimodal data comprising two or more of video data, audio data, or text data; processing, by the computing system, the context data corpus with one or more of the two or more pre-trained models to obtain a language-based context history, wherein the one or more pre-trained models include a language model; and The language-based context history and the data describing the query are processed by the computing system using the pre-trained language model of the two or more pre-trained models to obtain output data describing an answer to the query.

20. The method of claim 19, wherein the contextual data corpus includes video data and corresponding audio data; Wherein processing, by the computing system, the context data corpus with the one or more of the two or more pre-trained models comprises: Processing the video data by the computing system using a pre-trained visual language model from the one or more pre-trained models to obtain data describing a plurality of key frames of the video data; as well as The data describing the plurality of key frames of the video data is processed by the computing system using the pre-trained language model of the one or more pre-trained models to obtain the language-based context history.

Images