Backpack-based token optimization method and system in multi-modal large language model

Abstract

A method to improve computational efficiency of a multimodal large language model (MLLM) by reducing the number of tokens / embeddings processed by the neural network of the MLLM. Specifically, multimodal inputs are encoded into a shared embedding space so that embeddings can be compared across modalities. Furthermore, embeddings to be fed into the neural network are selected to: 1) maximize relevance of the selected embeddings with respect to a query; 2) maximize diversity within the selected embeddings (reduce redundancy); 3) maintain a total cost within a budget, the total cost reflecting computational cost associated with having the neural network process the embeddings. The proposed technique improves computational efficiency by reducing computational overhead of the neural network. Furthermore, the proposed technique preserves accuracy of the MLLM by preserving relevance of the selected embeddings with respect to the query.

Description

Technical Field

[0001] This application relates to the field of large model technology, and in particular to a knapsack-based lexical optimization method and system for multimodal large language models. Background Technology

[0002] Multimodal Large Language Models (MLLMs) are capable of processing and understanding multiple types of input data simultaneously, representing a significant advancement in artificial intelligence (AI). These models build upon traditional language models, integrating the ability to process images, videos, audio, and other modalities, as well as text, enabling them to form rich contextual understandings of different forms of information.

[0003] In the specific field of video question answering (VideoQA), MLLM has become the state-of-the-art method. MLLM has demonstrated exceptional capabilities in understanding video content and responding to user queries. These models process video frames sequentially, analyze the accompanying audio track, and interpret textual information such as subtitles or transcripts to form a comprehensive understanding of the video content. This enables MLLM to answer complex questions about events, relationships, and context within the video.

[0004] However, MLLM faces significant technical challenges, particularly in computational efficiency and lexical management. The multimodal nature of these models requires processing and storing a large number of lexical units for each modality, such as visual lexical units for video frames, audio lexical units for sounds, and text lexical units for queries and responses. This incurs substantial computational overhead and often exceeds the context window limit, necessitating fine-tuning of lexical usage across modalities while maintaining model performance.

[0005] It would be advantageous to implement an architecture and process for using and implementing MLLMs that at least partially address some of these problems. Summary of the Invention

[0006] This application proposes a technique to improve the computational efficiency of MLLM by reducing the number of terms / embeds processed by the neural network. Specifically, multimodal inputs are encoded into a shared embedding space, enabling cross-modal comparisons of the embeddings. Furthermore, the embeddings to be fed into the neural network can be selected to: 1) maximize the relevance of the selected embeddings to the query; 2) maximize the diversity within the selected embeddings (reducing redundancy); and 3) maintain the total cost within budget, reflecting the computational cost associated with enabling the neural network to process the embeddings. The proposed technique improves computational efficiency by reducing the computational overhead of the neural network. Furthermore, the proposed technique maintains the accuracy of the MLLM by preserving the relevance of the selected embeddings to the query.

[0007] In one aspect of the invention, a computer-implemented method is provided for generating output using a multimodal model. The method includes: receiving a query and associated multimodal input data. The multimodal input data includes at least two modalities. The method further includes: for each modality of the multimodal input data, encoding the modality of the input data using a corresponding encoder for that modality to generate a plurality of embeddings, the plurality of embeddings being mapped to a shared embedding space containing embeddings for all the modalities of the input data. Furthermore, the query is encoded into the shared embedding space as a query embedding. Additionally, a subset of the embeddings in the shared embedding space is selected. The subset is selected based on: 1) a distance measurement between the query embedding and a corresponding embedding in the shared embedding space; 2) a mapping of each embedding in the shared embedding space to a corresponding cost. The corresponding cost is based on the modality of the embedding. After selecting the subset, the selected subset of the embeddings and the query embedding are provided to a neural network in the multimodal model to generate a response to the query.

[0008] In some implementations, the distance measurement result is based on at least one of cosine similarity and L2 distance.

[0009] In some implementations, the method further includes: receiving at least one parameter; and defining the mapping based at least in part on said at least one parameter.

[0010] In some implementations, the embedded cost is defined by the hyperparameters of the multimodal model.

[0011] In some implementations, the corresponding cost of the embedding is defined based on the computational cost associated with the neural network processing the embedding.

[0012] In some implementations, the computational cost is based on at least one of memory usage, runtime, and processing power.

[0013] In some implementations, the subset is selected to maximize a value function. The value function at least measures the relevance of the subset relative to the query embedding. The subset is also selected to keep the total cost within budget. The total cost is obtained by: 1) obtaining a range by applying the mapping to the subset; 2) calculating at least one of a sum and an average over the range.

[0014] In some implementations, the method further includes receiving at least one parameter by user selection before selecting the subset. The budget is then determined based on the at least one parameter.

[0015] In some implementations, the value function includes a covering function that measures the relevance of the subset relative to the query embedding. The covering function measures the relevance at least by: 1) determining a first distance measurement between the first embedding and the query embedding for a first embedding in the subset; 2) determining a second distance measurement between the second embedding and the query embedding for a second embedding in the subset; and 3) applying a computational operation to the first and second distance measurements. The computational operation is one of a summation operation and an averaging operation.

[0016] In some implementations, the value function includes a reward function that biases the value function towards the diversity of embeddings. The reward function applies this bias based at least on an internal distance measurement between a first embedding and a second embedding in the subset.

[0017] In some implementations, selecting the subset further includes: obtaining copies of the plurality of embeddings; identifying marginal embeddings in the copies; the marginal embeddings maximizing the marginal gain per unit cost in the value function; the unit cost being calculated by applying the mapping to the marginal embeddings; adding the marginal embeddings to the selected embedding set; and also removing the marginal embeddings from the copies.

[0018] In some implementations, the at least two modalities may include at least one of text data, audio data, and visual data.

[0019] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium including one or more instructions, which, when executed on a computer, perform any of the methods disclosed herein.

[0020] According to another aspect of the invention, a non-transitory computer-readable medium is provided that stores instructions that enable a processor in a device to implement any of the methods disclosed herein.

[0021] According to another aspect of the invention, an apparatus is provided for performing any of the methods disclosed herein.

[0022] According to another aspect of the invention, a processor is provided that executes instructions to cause a device to perform any of the methods disclosed herein.

[0023] According to another aspect of the invention, an integrated circuit is provided for performing any of the methods disclosed herein.

[0024] According to another aspect of the present invention, a module is provided, the module comprising: one or more circuits for performing any of the methods disclosed herein.

[0025] According to another aspect of the invention, an apparatus is provided comprising: one or more processors, the one or more processors being functionally connected to one or more memories for performing any of the methods disclosed herein.

[0026] According to another aspect of the invention, an apparatus is provided for performing any of the methods disclosed herein.

[0027] In some embodiments, the apparatus includes one or more units for performing the methods described above.

[0028] According to another aspect of the invention, one or more non-transitory computer-readable storage media are provided, the one or more non-transitory computer-readable storage media comprising computer-executable instructions that, when executed, cause at least one processing unit, at least one processor, or at least one circuit to perform any of the methods disclosed herein.

[0029] According to another aspect of the invention, one or more computer-readable storage media are provided that store a computer program, which, when executed by a device, enables the device to implement any of the methods disclosed herein.

[0030] According to another aspect of the present invention, a computer program product is provided, the computer program product comprising one or more instructions that, when executed by a device, enable the device to implement any of the methods disclosed herein.

[0031] According to another aspect of the invention, a computer program is provided that, when executed by a computer, enables an apparatus to implement any of the methods disclosed herein.

[0032] According to another aspect of the invention, a system is provided that includes nodes for performing any of the methods disclosed herein. Attached Figure Description

[0033] The following is a detailed description of the embodiments with reference to the accompanying drawings: Figure 1 A computing environment for the disclosed embodiments is shown; Figure 2 A simplified diagram of the architecture used for optimizing MLLM lexical / embedding is shown; Figure 3 Another simplified diagram of another architecture used to optimize MLLM is shown; Figure 4A An exemplary computing system according to an exemplary embodiment is shown; Figure 4B An MLLM system with optimized use of lexical / embedding is shown; Figure 5 A flowchart illustrating the method for generating output using MLLM with optimized lexical / embedding is shown. Figure 6 The algorithm used for optimizing MLLM's lexical / embedding is shown; Figure 7 A flowchart of another method for generating output using MLLM with optimized lexical / embedding is shown.

[0034] The same reference numerals are used in the accompanying drawings to denote the same elements and features. Detailed Implementation

[0035] The embodiments disclosed herein relate to systems and apparatuses using MLLM and large language model (LLM). The systems and apparatuses disclosed herein may include suitable modules and / or circuitry for performing various processes.

[0036] As understood by those skilled in the art, "module" is an interpretative term referring to a hardware structure, such as circuitry, implemented using techniques such as electrical and / or optical technologies (and more specific examples of semiconductors) for performing defined operations or processes. Alternatively, "module" can refer to a combination of hardware and software structures, which are implemented using techniques such as electrical and / or optical technologies (and more specific examples of semiconductors) in a general manner for performing defined operations or processes according to the software structure in the form of an instruction set stored in one or more non-transitory computer-readable storage devices or media.

[0037] A module can be part of a device, apparatus, system, etc., and can be coupled to or integrated with other parts of the device, apparatus, or system so that their combination forms the device, apparatus, or system. Alternatively, a module can be implemented as a standalone device or apparatus.

[0038] Modules typically execute procedures used to perform methods. Here, "procedure" has the general meaning equivalent to "method." More specifically, a procedure is a defined method implemented using hardware components that process data. A procedure may include one or more functions or use one or more functions to process data as designed. Here, a function is a defined sub-procedure or sub-method used to compute, calculate, or otherwise process input data in a defined manner and to generate or otherwise produce output data.

[0039] As those skilled in the art will understand, the process can be implemented as one or more software and / or firmware programs having the necessary computer-executable code or instructions, and stored in one or more non-transitory computer-readable storage devices or media. These one or more non-transitory computer-readable storage devices or media can be any volatile and / or non-volatile, non-removable or removable storage device, such as RAM, ROM, EEPROM, solid-state storage devices, hard disks, CDs, DVDs, flash memory devices, etc. The module can read the computer-executable code from the storage device and execute the computer-executable code to perform the process.

[0040] Alternatively, the process can be implemented as one or more hardware structures with the necessary electrical and / or optical components, circuits, logic gates, integrated circuit (IC) chips, etc.

[0041] Machine Learning (ML), Neural Networks, LLM, and MLLM To help understand this invention, some concepts related to ML, neural networks, LLM, and MLLM will be discussed first.

[0042] Typically, a neural network consists of multiple interconnected artificial neurons arranged in multiple layers, including input layers, output layers, and often hidden (intermediate) layers. Neural networks are designed to process input data, extract relevant features, and generate outputs based on learned patterns and correlations. Artificial neurons in the network receive one or more input signals, apply weighted transformations to these inputs, and use activation functions to process the transformed data. Outputs from one layer in the network can be forwarded (input) to the next layer in the network, up to the final output layer of the neural network. Some neural networks may have more complex mechanisms for feeding data from one layer to another, rather than taking an output from one layer and providing that output to the next. For example, in some neural networks, skip connections can be used to feed the output of the first layer of the neural network to a second layer that is not immediately following the first layer. In another example, feedback connections can be used to feed the output of one layer in the neural network to the previous layer. Neural networks employ adaptive learning mechanisms, where the weights associated with the connections between neurons are iteratively adjusted based on a predefined training algorithm (e.g., backpropagation using gradient descent or alternative optimization techniques).

[0043] Neural networks can be trained using one or a combination of learning methods (e.g., supervised learning, unsupervised learning, reinforcement learning, etc.). Implementing a learning method (or a combination of methods) allows the neural network to improve its performance over time by minimizing an error function or optimizing an objective criterion. Examples of supervised learning include, but are not limited to, Naive Bayes algorithms, regression algorithms (e.g., linear, nonlinear, logistic, polynomial, etc.), support vector machines (SVM), K-nearest neighbor (KNN), random forest algorithms, and deep learning algorithms. Examples of unsupervised learning include, but are not limited to, K-means clustering, hierarchical clustering, density-based spatial clustering of applications with noise (DBCAN), principal component analysis (PCA), t-distributed stochastic neighbor embedding (t-SNE), autoencoders, isolated forests, a class of SVM algorithms, the Apriori algorithm, the Eclat algorithm, generative adversarial networks (GANs), and deep learning algorithms. Examples of reinforcement learning include, but are not limited to, Q-learning, actor-critic algorithms, and policy iteration algorithms.

[0044] Applications of neural networks include, but are not limited to, image and speech understanding and generation, natural language processing (NLP), predictive analytics, anomaly detection, and autonomous decision-making systems. The architecture of a neural network can be customized by changing the number of layers, the type of activation function (e.g., sigmoid function, Tanh activation function, rectified linear function, softmax function, softplus function, etc.), the connection pattern (e.g., skip connections, feedback connections, etc.), and the training method.

[0045] Deep neural networks (DNNs) consist of multi-layered neural networks used for high-dimensional data processing, feature extraction, and predictive modeling. Examples of DNNs include, but are not limited to, convolutional neural networks (CNNs), recurrent neural networks (RNNs), multilayer perceptrons (MLPs), and combinations thereof. ML models are often implemented using DNNs.

[0046] Training an ML model typically involves a series of steps. Training can begin with acquiring raw input data, which may undergo preprocessing steps such as data normalization, feature scaling, dimensionality reduction, missing value imputation, and augmentation techniques. In supervised learning, the labeled dataset can be divided into training, validation, and test sets. In unsupervised learning, clustering or transformation methods can be applied to the raw input data. The parameters of the ML model can then be initialized. For example, the weights and biases associated with the artificial neurons in the neural network can be initialized to apply weighted transformations. In some implementations, parameters can be initialized using randomization, pre-training, or heuristic-based initialization techniques. During training iterations, the ML model processes sample inputs, where each computational unit (artificial neuron) undergoes a weighted transformation and then an activation function is applied. The model can then generate an output that is compared to the expected result in supervised learning, or, in the case of supervised or reinforcement learning, evaluated using an objective function. Loss functions (e.g., mean squared error (MSE), cross-entropy loss, reinforcement learning reward functions, etc.) quantify the difference between the ML model's output and the expected result. To improve ML models (i.e., to make the output closer to the desired output), optimization techniques such as stochastic gradient descent (SGD), adaptive moment estimation (Adam), and root mean square propagation (RMSprop) are used to tune the parameters of the ML model. In some implementations, optimization techniques may involve backpropagation, where the gradient of the loss function is computed relative to the parameters of the ML model using the differential chain rule. These gradients guide the parameters to be updated in the direction that minimizes the loss. To prevent overfitting and enhance model generalization, various regularization techniques can be applied during training. For example, L1 and / or L2 regularization can be used to penalize the loss function for high correlation coefficients. Other regularization techniques include, but are not limited to, data augmentation, early stopping, dropout regularization, and batch normalization.

[0047] The training process involves monitoring the performance of the ML model using evaluation metrics such as accuracy, precision-recall, F1 score, AUC-ROC, mean absolute error (MAE), and combinations thereof. Calculating these evaluation metrics may involve processing a validation set using the ML model. Furthermore, based on the evaluation metrics, the hyperparameters of the ML model (e.g., the number of layers in the neural network, the learning rate, etc.) can be fine-tuned. After fine-tuning the hyperparameters of the ML model, a new iteration of the training process can be performed. The training process may terminate when the improvement of the ML model plateaus.

[0048] In some implementations, after the training process is complete, the ML model can be finally evaluated using a test set. Evaluation metrics can be used to assess the performance of the ML model.

[0049] Once the training process is complete, the parameters and hyperparameters of the ML model can be fixed and deployed for use.

[0050] In some implementations, for example, the trained ML model can be fine-tuned for a specific task. That is, the parameters of the ML model can be adjusted to improve its performance relative to a specific task. The fine-tuning process of the ML model is similar to the training process described in this paper. Typically, the dataset used for fine-tuning the ML model is smaller than the dataset used to train the ML model.

[0051] LLMs are DNN-based AI systems designed for NLP, text generation, context understanding, and multimodal interaction. LLMs can be implemented using neural network architectures known as Transformers. Examples of Transformer-based LLMs include, but are not limited to, Bidirectional Encoder Representations from Transformer (BERT) models, Transformer-XL models, and Generative Pre-trained Transformer (GPT) models. Transformers can leverage the parallel processing capabilities of graphics processing units (GPUs), neural processing units (NPUs), and / or tensor processing units (TPUs) to efficiently process sequential data, enabling them to perform computations efficiently.

[0052] Transformer-based LLMs can perform tokenization on inputs to the LLM (e.g., text prompts). That is, the LLM can generate a sequence of lexical units (or multiple lexical units) based on the input. In some embodiments, the LLM can use an encoder or encoding module to generate lexical units. In one example, if the input to the Transformer is a sequence of words such as "the quickbrown fox," the LLM can generate one lexical unit for each word. That is, the LLM can generate a first lexical unit corresponding to "the," a second lexical unit corresponding to "quick," a third lexical unit corresponding to "brown," and a fourth lexical unit corresponding to "fox." Furthermore, the encoder can generate special lexical units that encode contextual information, such as [CLS] lexical units. Lexical units can be thought of as numerical representations of the corresponding words. Therefore, tokenizing the input to the LLM may produce a numerical sequence corresponding to that input.

[0053] After word segmentation, a Transformer-based LLM can convert each terminator into an embedding (or vector) in an embedding space (or vector space). This embedding can be performed by an embedding layer, an embedding module, or a projector. In some implementations, such as those using an embedding layer, embedding can be performed within the Transformer architecture. That is, the embedding layer is a layer within the Transformer. In these implementations, embedding is performed based on the parameters of the Transformer, and the mapping from terminator to embedding can change or evolve as the LLM is trained. In other implementations, the LLM can convert terms into embeddings before feeding the corresponding data to the Transformer. In these implementations, the terms "terminator" and "embedding" can be used interchangeably, where applicable. Converting terms into embeddings allows the LLM to have similar mathematical representations for semantically similar terms (e.g., "rain" and "precipitation").

[0054] A Transformer may include encoder layers / blocks. Encoder layers / blocks may output embeddings (in an embedding space), vector representations (in a vector space), or features (in a feature space) in response to received input. In some implementations, encoder layers / blocks may output embeddings based on weighted transformations. Examples of encoder layers / blocks include the embedding layers discussed earlier. Another example includes a positional encoder that adds positional information related to the placement of the original lexical units in the original lexical sequence to the embeddings in the embedding space. Encoder layers / blocks may include a series of connection layers within a Transformer.

[0055] A Transformer can also include decoder layers / blocks. Decoder layers / blocks can take embeddings, vector representations, or features (e.g., the output of encoder layers / blocks) as input and generate outputs. For example, given embeddings corresponding to a sequence of words, a decoder layer / block can generate outputs (e.g., lexical units) corresponding to the next word in that sequence. In some implementations, decoder layers / blocks can feed their outputs back to themselves to generate another output. That is, decoder layers / blocks can continue generating output (lexical) sequences by feeding their outputs back to themselves. In this example, the decoder layer / block can output a specific lexical unit, such as the [EOS] lexical unit. [EOS] indicates the end of the sequence. After generating the [EOS] lexical unit, the decoder layer / block can stop generating outputs. Decoder layers / blocks can include a series of connection layers in a Transformer.

[0056] It should be understood that a Transformer can include more than one encoder layer / block and more than one decoder layer / block. Some Transformers are implemented using both encoder and decoder layers / blocks. Such Transformers are used to implement the Text-to-Text Transformer (T5) model. Some Transformers (e.g., BERT) are implemented using only encoder layers / blocks. Furthermore, some Transformers can be implemented using only decoder layers / blocks. The GPT model can be considered as such an implementation of the Transformer.

[0057] Transformers or Transformer-based LLMs can also include generators or generation modules. A generator can transform the output of a decoder layer / block (e.g., a sequence of terms) into the final output of the LLM. For example, if a decoder layer / block outputs a sequence of terms corresponding to the text "jumps over the lazy dog," the generator can generate a sequence of words corresponding to "jumps over the lazy dog."

[0058] MLLM is an AI system capable of processing and integrating data from multiple modalities, including text, images, audio, and video, to generate output or perform tasks requiring cross-modal understanding. An example of a task requiring cross-modal understanding is generating text descriptions for video clips or providing video-related answers in response to a user query. MLLM can employ combinations of deep learning architectures.

[0059] MLLMs can use modality-specific encoders / projectors to generate lexical units and / or embeddings. For example, a text encoder can be used to generate lexical units / embeddings corresponding to text input. Similarly, an image encoder can be used to generate embeddings corresponding to visual data such as video frames. The generated lexical units / embeddings can then be fed into a neural network (e.g., an LLM) to generate data or perform a task.

[0060] Optimize lexical / embedding usage and knapsack problem To aid in understanding this invention, some concepts related to MLLM and lexical / embedding optimization will be discussed.

[0061] In specific embodiments, MLLM can process a group (or more) of shared embedding spaces (represented as...). The embeddings, a set (or more) of embeddings, are obtained based on lexical units generated by processing multimodal input data such as video (e.g., frames, audio, text transcription, and metadata). Furthermore, each modality of the multimodal input data may have an associated cost. Therefore, each of the embeddings (lexical units) in this set may also be associated with a cost. The cost of each lexical / embedding (modality) may reflect the computational cost associated with processing that lexical / embedding in an LLM (MLLM) within a neural network. Additionally, the MLLM can also process queries associated with the multimodal input data and generate query embeddings, represented as... Query embeddings also reside in the shared embedding space. That is to say, and All elements in the shared embedded space are located in the shared embedded space. The relevance of the embeddings may depend on Therefore, the goal is to improve the computational efficiency of MLLM by reducing the number of embeddings / terms processed by the neural network. Specifically, the objective is to select a subset. The relevance of the embeddings in the subset is maximized, and the total cost associated with the subset is minimized (or below the budget or threshold). This problem can be described as a knapsack problem, where the goal is to maximize the value in a set given constraints (e.g., budget). Therefore, choosing the subset can be formulated as obtaining the value defined by the following formula. subset of :

[0062] in It is a value function, given input In this case, the value function outputs the value. It is embedded The cost, This is the budget that the total cost associated with the selected subset cannot exceed. In this case, the budget can be considered the computational cost or the maximum computational value.

[0063] In addition, value functions It can be defined by the following formula:

[0064] in It is a "coverage" function that measures the input set. right The degree of coverage, It is a "reward" function that rewards diversity among the selected embeddings and / or penalizes redundancy among the selected embeddings. These are the parameters to be weighed. Function This can be understood as measuring the input set. Compared to (And further extended to the query itself) a function of relevance.

[0065] It can be the sum of the shift cosine similarities between the embedding and the query. Specifically, It can be defined according to the following formula:

[0066] in It is a function that calculates the cosine similarity between embeddings. This can be understood as... Measure subsets at least in the following ways The relevance is as follows: 1) For the first embedding in the subset, determine the first distance measurement between the first embedding and the query embedding; 2) For the second embedding in the subset, determine the second distance measurement between the second embedding and the query embedding; 3) Summate the first distance measurement and the second distance measurement. It should be understood that, although as defined herein, This is defined by applying a summation to multiple distance measurement results, but in other embodiments, a similar coverage function can use an averaging operation. Therefore, step 3) above can be summarized as: 3) applying a calculation operation to the first distance measurement result and the second distance measurement result, which is one of a summation operation and an averaging operation. It should also be understood that... The value varies with the input set Embedding and The distance between them decreases and increases. Therefore, maximizing (partially) defines of The selection biases towards embeddings within a subset that are closer to... In other words, a subset can be selected. Maximize the value function, which at least measures the subset relative to the query embedding. The correlation.

[0067] It can be based on the input set Reward / penalize the maximum distance between elements This encourages Internal diversity. Specifically, It can be defined according to the following formula: .

[0068] It should be understood that, The value varies with the input set The embeddings in the matrix increase as the distance between them increases. Therefore, maximizing (partially) defines... of It favors selecting subsets where the embeddings are far apart (or less redundant). In other words, Value function It leans towards diversity in embedding. It can be considered that... Based on subsets The first embedding and subset in The internal distance measurement results between the second embeddings in the middle Apply bias.

[0069] It should be understood that, although as defined in this article, , and The distance measurement result is defined using cosine similarity, but in other embodiments of lexical / embedding optimization, distance measurements based on other types of distance metrics can be used to define similar functions for similar purposes. For example, another coverage function could be based on L2 distance. Therefore, it can be considered that for The selection of equal subsets can be based on distance measurements, which are based on at least one of cosine similarity and L2 distance.

[0070] It should also be noted that, as defined in this article, It is a monotonic submodular function. A submodular function is a function that outputs a value given an input set, and adding elements to that input set typically provides diminishing marginal returns. A monotonic function is a function that outputs a value given an input set, and adding elements to that input set either preserves or increases the output value. A monotonic submodular function is both a submodular function and a monotonic function.

[0071] It can be considered as embedding A function or mapping that maps to the associated cost. Specifically, It can be defined according to the following formula:

[0072] in These are the hyperparameters of MLLM, designed to reflect the computational cost associated with the neural network processing text, audio, and image modalities respectively, and are therefore typically set to make such that... .

[0073] In other words, the corresponding cost of an embedding can be defined based on the computational cost associated with the neural network processing that embedding. This computational cost can be based on at least one of memory usage, runtime, and processing power. It should also be noted that by... Represented as a mapping, subsets can be obtained in the following ways. Total cost: 1) Through the analysis of 1) Apply the mapping to obtain the range; 2) Calculate the sum over that range. It should be understood that, although as defined in this paper, the knapsack problem maintains its properties through mapping the range to the range of knapsack problems, the knapsack problem is not fully understood. The total cost is calculated by summing the costs associated with the selected embeddings. However, in other embodiments involving word / embedding optimization for solving similar knapsack problems, the total cost can be calculated as the average cost associated with the selected embeddings. Therefore, step 2) above can be summarized as: 2) calculating at least one of the sum and average over this range.

[0074] This invention will now continue to discuss the selection and / or pruning of terms to improve the computational efficiency of MLLM.

[0075] MLLM is implemented through optimization of lexical / embedding usage. To improve computational efficiency, some MLLMs may be designed to perform lexical reduction. Some MLLMs can perform lexical reduction through lexical pruning techniques. In these techniques, lexical units (e.g., visual units corresponding to the image modality) can be pruned after the segmentation process. These techniques typically use analysis of attention score metrics to identify and prune lexical units considered less important to model performance. Other MLLMs can perform lexical reduction by performing keyframe selection techniques. In these techniques, redundant or unnecessary frames can be identified and removed through embedding similarity analysis, thereby selecting / pruning lexical units (e.g., visual units corresponding to the image modality) before the segmentation process. The disadvantages of these techniques include: 1) the pre-training process is computationally expensive and may degrade performance; 2) reliance on external corpora, including knowledge base training sets, which increases resource requirements and limits mobile applications; 3) the need to modify the model architecture or customize neural subnetworks, thus reducing broad applicability; and 4) limited focus on image modality optimization, thus ignoring potential lexical reduction in other modalities and cross-modal redundancy. In other words, these techniques involve modifying the architecture of MLLM, increasing computational complexity in resource-constrained environments, limiting applicability to mobile applications, and missing opportunities to leverage complementary information between modalities.

[0076] As will be described herein, this invention describes systems and methods for improving the computational efficiency of MLLMs by reducing the number of terms / embeddings processed by the neural network of the MLLM. Specifically, these systems and methods involve: 1) embedding multiple modalities into a shared embedding space, thereby enabling cross-modal comparisons between terms / embeddings corresponding to different modalities; 2) reducing redundant terms / embeddings processed by the neural network by measuring the relevance of the embeddings in the embedding space to the query; and 3) using a minimal term modality selector that maximizes relevance to the query while minimizing the total number of terms / embeddings processed by the neural network. The advantages of the described systems and methods include, but are not limited to: 1) the absence of a costly pre-training process that could potentially degrade model performance; 2) independence from external datasets; 3) no need to modify the neural network architecture; and 4) leveraging complementary information between modalities to reduce redundancy between terms / embeddings of different modalities.

[0077] refer to Figure 1 This figure illustrates an exemplary embodiment of a computing environment for providing / deploying lexical / embedding optimizations to improve computational efficiency in MLLMs. Figure 1 The diagram shows a computer system 110, a database 120 coupled to the computer system 110, a computing device 130, and a network 140 connecting the computer system 110 and the computing device 130.

[0078] Computer system 110 (depicted as a server) may be a computer system associated with maintaining one or more models 122 stored in database 120. The one or more models 122 may include at least one MLLM. Furthermore, the MLLM may enable VideoQA. Computer system 110 may access, train, use, or modify model 122. In some embodiments, database 120 may be integrated into computer system 110 or located within the computer system. That is, computer system 110 may access model 122 from internal storage. In other embodiments, database 120 may be remote from the computer system (e.g., a cloud server) and / or located outside the computer system. That is, computer system 110 may access model 122 from remote and / or external storage devices.

[0079] Computing device 130 (depicted as a laptop computer) can be a computer used by a client. Specifically, computing device 130 or an associated client can access or use model 122 through a connection to computer system 110. For this purpose, computing device 130 can communicate with computer system 110 using an application programming interface (API). In an exemplary scenario, computing device 130 can transmit cue and video data to computer system 110 via network 140. For example, the cue could be “Generate a text summary of the provided document”. Computer system 110 can then pass the cue and video data to an MLLM included in model 122. The MLLM can then generate a text summary as output. In this example, computing device 130 can also provide a pruning ratio to computer system 110. Given a pruning ratio, computer system 110 can cause the MLLM to reduce lemmas / embeddings based on the received pruning ratio to improve computational efficiency. Additionally or alternatively, computing device 130 can provide computer system 110 with custom hyperparameters that can be used to reduce lemmas / embeddings. For example, computing device 130 can provide... , and The value, or alternatively, a value that enables computer system 110 to determine. , and The value of the value. Then, computer system 110 can transmit the summary to computing device 130. In this exemplary scenario, computing device 130 may be associated with a client that has signed a service agreement with an entity associated with computer system 110. For example, the service agreement may be a service agreement for accessing and using MLLM used in the described exemplary scenario.

[0080] It should be understood that, although Figure 1A single computing device 130 is shown, but in reality, there may be multiple computing devices and associated clients connected to the computer system 110 to access and use model 122.

[0081] Figure 1 A network 140 is shown connecting computer system 110 and computing device 130. Network 140 may be, but is not limited to, a wireless network, radio network, cellular network, intranet, Internet network, 5G network, Wi-Fi network, local area network, wide area network, or a combination thereof. Computer system 110 and computing device 130 can send data and / or requests to each other through network 140.

[0082] refer to Figure 2 , Figure 2 A simplified diagram illustrating the architecture used for optimizing MLLM 200's lexical / embedding is shown. MLLM 200 can be a computer system 110 (see...). Figure 1 Models for maintaining computer systems, such as computers.

[0083] Figure 2 The MLLM 200 receives inputs. Specifically, the MLLM 200 is shown as receiving a query 210 and a multimodal input 212. The query 210 may be from, for example, a computing device 130 (see...). Figure 1 The text query is received by a computing device such as a computing device. Similarly, multimodal input 212 can also be received from a computing device. Multimodal input 212 can include data from at least two modalities. That is, multimodal input 212 can be divided into at least a first modal input 222 and a second modal input 224. Examples of modalities that can correspond to the first modal input 222 or the second modal input 222 include, but are not limited to, audio, text, and frames (e.g., video frames). Additionally or alternatively, query 210 can be associated with multimodal input 212. For example, query 210 could be a prompt “identify the part of this lecture that discusses lexical reduction,” and multimodal input 212 could be a recording and corresponding text transcription of a lecture related to lexical reduction in MLLM. That is, the first modal input 222 could be audio data from the recording, and the second modal input 224 could be a text transcription of the recording.

[0084] Figure 2 Also shown is an MLLM 200 including a text encoder 230, a first encoder 232, and a second encoder 234. The text encoder 230 transforms the query 210 into a query embedding in a shared embedding space. That is, the text encoder can be considered as transforming the query 210 and the output... As input, the first encoder 232 and the second encoder 234 are encoders specific to the first modality and the second modality, respectively. The first encoder 232 can take the first modality input 222 as input and output embeddings in a shared embedding space. Similarly, the second encoder 234 can take the second modality input 224 as input and output embeddings in the shared embedding space. In some embodiments, the first encoder 232 and the second encoder 234 can be components of a modality embedding block. The modality embedding block can include multiple modality-specific encoders that map inputs to a shared embedding space. Since the query 210, the first modality input 222, and the second modality input 224 are mapped to the shared embedding space, various inputs can be compared with each other for lexical / embedding reduction purposes. For example, the first modality input 222 and the second modality input 224 can be compared with the query 210 to measure the relevance of the lexical / embeddings of the first modality input 222 and the second modality input 224 to the query 210. In addition, the lexical units / embeddings of the first modal input 222 and the second modal input 224 can be compared with each other to reduce cross-modal redundancy between lexical units / embeddings.

[0085] It should be understood that text encoder 230, first encoder 232, and second encoder 234 can perform the following two operations: 1) segmenting the query 210, first modal input 222, and second modal input 224; 2) outputting / generating embeddings in a shared embedding space based on the lexical units generated by the segmentation process. It should also be understood that an embedding can correspond to a lexical unit (e.g., since lexical units can be converted into embeddings), therefore reducing lexical units can correspond to reducing embeddings, and vice versa. Therefore, where applicable, the terms "lexical unit" and "embedded unit" are used interchangeably throughout this disclosure.

[0086] Figure 2 Also shown is an MLLM 200 including a minimum lexical modality selector 240. The minimum lexical modality selector 240 can select the embeddings (output by the first encoder 232 and the second encoder 234) to be processed by the LLM 250. Specifically, the minimum lexical modality selector 240 can select embeddings in order to: 1) maximize the relevance of the selected embeddings to the query 210 (or its corresponding embedding output by the text encoder 230); 2) minimize information redundancy among the selected lexicals; and 3) keep the total cost below budget. The minimum lexical modality selector 240 can be considered as selecting lexicals by solving the knapsack problem described herein.

[0087] After selecting the embedding in the minimum token modality selector 240, the query embedding is retrieved. The embeddings selected by the text encoder 230 (output by the text encoder 230) and the minimum lexical modality selector 240 can be fed into the LLM 250 to generate output 260. In the example where query 210 is “identify the portion of this lecture that discusses lexical reduction”, output 260 could be a text sequence identifying the start and end times in the recording of the lecture. For example, output 260 could be “00:36:21 to 00:43:38”.

[0088] refer to Figure 3 , Figure 3 A simplified diagram illustrating the architecture used for optimizing MLLM 300's lexical / embedding is shown. MLLM 300 can be a computer system 110 (see...). Figure 1 Models for maintaining computer systems such as MLLM300. As will be described, MLLM300 is similar to MLLM200 (see MLLM200). Figure 2 In fact, MLLM 300 can be considered a specific implementation of MLLM 200.

[0089] Figure 3 The MLLM 300 is shown receiving inputs. Specifically, the MLLM 300 is shown receiving a query 310 and a video 312 as inputs. The query 310 can be similar to the reference. Figure 2 The description of query 210. Video 312 is similar to the references in this article. Figure 2 The multimodal input 212 is described. Similar to query 210 and multimodal input 212, query 310 and video 312 can be associated. For example, query 310 could be "What is the license plate of the fastest car in this video clip?", while video 312 could be the corresponding video clip.

[0090] Figure 3 The video 312 is shown to be divided into its various modalities, namely frame 322, audio 324, and text (text transcription 326 and metadata 328). Frame 322, audio 324, text transcription 326, and metadata 328 can be referenced herein. Figure 2 The first modal input 222 and the second modal input 224 described are equivalent.

[0091] Figure 3 A text encoder 330 is also shown, which outputs the query embedding in a shared embedding space as a result of processing query 310. The operation and function of the text encoder 330 can be similar to those described in the references herein. Figure 3 The text encoder 230 is described. Figure 3Image encoder 332, audio encoder 334, and text encoder 336 are also shown, which similarly output embeddings in a shared embedding space by processing inputs from corresponding modalities. The operation and function of image encoder 332, audio encoder 334, and text encoder 336 can be similar to those described herein. Figure 2 The first encoder 232 and the second encoder 234 are described. It should be understood that in some embodiments, the text encoder 330 and the text encoder 336 may be the same encoder.

[0092] Figure 3 Also shown is the in-memory semantic retrieval block 338. The in-memory semantic retrieval block 338 utilizes a vector database to efficiently store and retrieve embeddings. By employing cosine similarity, the in-memory semantic retrieval block 338 can quickly identify embeddings similar to the query embedding. This retrieval process can be crucial for accurately locating relevant content across different modalities, enabling the MLLM 300 to respond accurately and promptly to received inputs (query 310 and video 312).

[0093] Figure 3 Also shown is an MLLM 300, which includes a minimal lexical modality selector 340 for selecting the embeddings to be fed to the LLM 350. The operation and function of the minimal lexical modality selector 340 can be similar to those described herein. Figure 2 The minimal lexical modal selector 240 is described.

[0094] After selecting the embedding in the minimum token modality selector 340, the query embedding is retrieved. The embeddings selected by the text encoder 330 (output by the text encoder 330) and the minimum token modality selector 340 can be fed into the LLM 350 to generate output 360. In the example where query 310 is “What is the license plate of the fastest car in this video clip?”, output 360 could be a text representation of the license plate identified in video 312.

[0095] refer to Figure 4A , Figure 4A An exemplary computing system 400 is shown, which can be used to implement examples of the present invention, such as MLLM 300 (see [link]). Figure 3 Additionally or alternatively, one or more instances of the exemplary computing system 400 may be used to perform MLLM. For example, multiple instances of the exemplary computing system 400 may collaborate to use MLLM to provide output. Furthermore, the exemplary computing system 400 may be an example of computer system 110 or computing device 130 (see [link to documentation]). Figure 1 ).

[0096] An exemplary computing system 400 includes at least one processing unit (e.g., processor 402) and at least one physical memory 404. For example, processor 402 may be a central processing unit, microprocessor, digital signal processor, application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), dedicated logic circuit, dedicated AI processor unit, GPU, TPU, NPU, hardware accelerator, or a combination thereof. Memory 404 may include volatile or non-volatile memory (e.g., flash memory, random access memory (RAM), and / or read-only memory (ROM)). Memory 404 may be a non-transitory computer-readable medium storing instructions that are executed by processor 402 to computing system 400 to perform examples of the methods, functions, systems, and modules disclosed herein.

[0097] The computing system 400 may also include at least one network interface 406 for wired and / or wireless communication with external systems and / or networks (e.g., intranets, the Internet, P2P networks, WANs, and / or LANs). The network interface enables the computing system 400 to communicate (e.g., wirelessly) with systems outside the computing system 400 (e.g., language models residing on remote systems).

[0098] The computing system 400 may optionally include at least one input / output (I / O) interface 408, which can be connected to one or more optional input devices 410 and / or one or more optional output devices 412. The one or more input devices 410 may include, for example, buttons, microphones, touchscreens, keyboards, etc. The one or more output devices 412 may include, for example, displays, speakers, etc. In this example, the one or more optional input devices 410 and the one or more optional output devices 412 are shown as being located outside the computing system 400. In other examples, one or more of the input devices 410 and / or one or more output devices 412 may be internal components of the computing system 400.

[0099] refer to Figure 4B , Figure 4BAn exemplary MLLM system 430 is illustrated. The MLLM system 430 is used to generate outputs such as sequences (e.g., text) using LLM. According to embodiments currently disclosed, components of the MLLM system 430 can collectively implement various lexical / embedding reduction techniques. The MLLM system 430 includes a generation engine 440 and a selection module 460. The generation engine 440 processes user-provided input 420 (e.g., input prompts, text input, and visual input) and generates output 450 based on the input 420. The generation engine 440 is communicatively coupled to the selection module 460. According to embodiments described herein, the selection module 460 is used to reduce lexical / embeddings. In some embodiments, the generation engine may include a neural network including hidden layers, encoder layers, decoder layers, etc., of MLLM and / or LLM. In some embodiments, the selection module 460 may perform similar actions as described herein. Figure 2 and Figure 3 The functions of the minimum lexical modality selector 240 and the minimum lexical modality selector 340 are described.

[0100] refer to Figure 5 , Figure 5 A flowchart of method 500 for generating output using MLLM with optimized lexical / embedding is shown. Method 500 can be implemented by a computing device with suitable processor-executable instructions for causing the computing device to perform the operation. Method 500 can be implemented wholly or partially by computer system 110, computer system 400, or MLLM system 430 (see [link to documentation]). Figure 1 , Figure 4A and Figure 4B This can be implemented using computer systems such as computing devices (e.g., computing device 130) or associated clients. As part of method 500, the computer system can maintain the MLLM, and computing devices (e.g., computing device 130) or associated clients can access or use the MLLM by communicating with the computer system.

[0101] Method 500 begins with operation 510. At operation 510, the system receives a query and associated multimodal input data. The multimodal input data may include at least two modalities. In one specific embodiment, the at least two modalities may include at least one of text data, audio data, and visual data. An example of the query includes query 310, and an example of the multimodal input data includes video 312 (see [link to video]). Figure 3 In some embodiments, the query and the associated multimodal input data can be received from a computing device or an associated client. For example, computing device 130 can send the query and the multimodal input data to computer system 110.

[0102] Following operation 510, method 500 can proceed to operation 520. At operation 520, the system can encode the multimodal input data to generate multiple embeddings mapped to a shared embedding space. For each modality of the multimodal input data, the system can generate multiple embeddings by encoding that modality of the input data using the corresponding encoder for that modality. Furthermore, the shared embedding space can contain embeddings for all modalities of the multimodal input data. The system can use image encoder 332 and audio encoder 334 (see [link to image encoder]). Figure 3 Modality-specific encoders, such as those used in this document, are employed to perform operation 520. In some embodiments, the plurality of embeddings may be a set as described herein. .

[0103] Following operation 520, method 500 can proceed to operation 530. At operation 530, the system can encode the query into the shared embedding space as a query embedding within that shared embedding space. The system can use text encoder 330 (see...). Figure 3 The query embedding can be performed using a text encoder such as a text encoder. In some embodiments, the query embedding can be an embedding as described herein. It should be understood that, although Figure 5 Operations 520 and 530 are shown in the order that operation 520 precedes operation 530, but operations 520 and 530 can be performed in any order, simultaneously or nearly simultaneously.

[0104] Following operation 530, method 500 may proceed to operation 540. At operation 540, the system may select a subset of the embeddings in the shared embedding space. In some embodiments, the selected subset may be a subset as described herein. The system can select the subset based on: 1) the distance measurement between the query embedding and the corresponding embedding in the shared embedding space; and 2) the mapping of each embedding in the shared embedding space to its corresponding cost. This corresponding cost can be based on the modality of the corresponding embedding. Specifically, the subset can be selected to: 1) maximize the relevance of the embedding in the subset to the query (which can be expressed as minimizing the distance measurement between the embedding in the subset and the query embedding); 2) minimize the redundancy of embeddings within the subset; and 3) keep the total cost below budget. The following will refer to... Figure 6 Exemplary details related to the process of selecting this subset are described.

[0105] After operation 540, method 500 can proceed to operation 550. At operation 550, the system can provide the selected subset of the embedding and the query embedding to the neural network (e.g., LLM) in the MLLM to generate a response to the query.

[0106] After generating the response, the system can send the response to the computing device or the associated client. For example, if the computing device is using or accessing the MLLM and input data (the query and the multimodal input data) is received from the computing device for this purpose, the response can be sent to the computing device.

[0107] As described herein, method 500 includes selecting a subset of the embedding at operation 540 in order to: 1) maximize the relevance of the embedding in the subset to the query (which can be expressed as minimizing the distance measurement between the embedding in the subset and the query embedding); 2) minimize the redundancy of the embeddings within the subset; and 3) keep the total cost below budget. Reference will now be made to... Figure 6 An embodiment for selecting a subset is discussed, and the figure depicts an algorithm 600 for selecting an embedded subset.

[0108] As mentioned earlier, the goal of Algorithm 600 is to select a set by solving the knapsack problem. subset of ,in It is multiple embeddings (e.g., as referenced) Figure 5 As mentioned above, it is generated at operation 520 of method 500. That is to say:

[0109] in It is a value function, given input In this case, the value function outputs the value. It is embedded The cost, It is the budget that the total cost associated with the selected subset cannot exceed.

[0110] As mentioned above, It is a value function defined according to the following formula:

[0111]

[0112] .

[0113] In addition, as mentioned above, It can be defined according to the following formula: .

[0114] It should also be noted that Algorithm 600 is a greedy algorithm. Furthermore, given... It is a monotonic submodular function, and Algorithm 600 solves the knapsack problem under discussion approximately optimally.

[0115] Algorithm 600 starts from line 1. In line 1, the set... Set to an empty set. This can be understood as... The embeddings in [the algorithm] are added to the working set. As Algorithm 600 progresses, The embedding in is added In the middle, until Potentially becoming the desired .

[0116] After line 1, Algorithm 600 continues to line 2. In line 2, the set... Initialize to be equal In other words, obtaining multiple embeddings ( A copy of () ).and similar, This can be understood as the working set used for the purposes of Algorithm 600. (and) different, The embeddings will change as algorithm 600 progresses. Remove from the middle. Therefore, when... Initialize to During the copying process, as algorithm 600 proceeds, will gradually with different.

[0117] After line 2, the algorithm continues with a loop from lines 3 to 9. As can be seen from line 3, this loop will continue executing until... Empty. Once If the value is empty, the algorithm will continue to line 10. Within this loop, as shown in line 4, the embedding is obtained. ,in: .

[0118] In other words, make Maximizing the marginal gain per unit of cost. In other words, make Improvements and connections Maximize the ratio between costs. As shown in the formula above, marginal gain is achieved by... Add to Caused by The increase in output, divided by the amount obtained through the... Application Mapping Calculated The cost is calculated based on the cost of the copy. Line 4 can be understood as including the identification of the copy ( Marginal embedding in ) The operation of the marginal embedding makes the value function ( Maximize the marginal gain per unit cost in ). Furthermore, the unit cost can be maximized by applying a mapping to the marginal embedding ( ) to calculate.

[0119] After line 4, and still within the same loop, Algorithm 600 continues to lines 5 through 7. In lines 5 through 7, two conditions are checked. The first condition checked is... Add to Costs in )make Total costs remain within budget Inside. The total cost is achieved through the analysis of... The cost of each element in the equation is summed to calculate the result. The second condition being checked is that when applied to... At that time, Add to Will it increase or remain? The value of . In other words, the second condition check will be performed by . Add to obtained Whether the improvement in the value is non-negative. It should be understood that in some embodiments, it is not necessary to check the second condition. For example, if It is a monotonic function (which is what is defined as a function for this article). In the case of adding an element to a set, it will not reduce the effect of applying the set to the desired value. The obtained value. If both conditions are met, then in line 6... Add to In other words, embedding the margins ( Add to the selected embedding set ( )middle.

[0120] Check lines 5 through 7 and possibly Add to After that, the algorithm continues to line 8. In line 8, it will... from Remove from the middle. That is, embed the margin ( From copy ( Remove from ) At this time, if If the value is not empty, Algorithm 600 continues to the beginning of the loop and executes line 4 again. That is, Algorithm 600 executes another iteration of the loop. On the other hand, if... If the value is empty, the algorithm continues to line 10.

[0121] On line 10, get the embedding. . Specifically: .

[0122] In other words, yes Embedding in, its 1) in the Application of single-element subsets Maximize time The value of , and 2) cost In budget Inside. It can be understood as The single best option in the list.

[0123] After line 10, Algorithm 600 continues to line 11. In line 11, it... and application Then, Set to when input The larger of the two output values.

[0124] In general, Algorithm 600 can be understood as performing subset operations in at least the following ways. Choice: 1) Obtain multiple embeddings ( A copy of () ); 2) Identify the marginal embeddings in the copy. This marginal embedding maximizes the marginal gain per unit cost in the value function.

[0125] refer to Figure 7 , Figure 7 A flowchart of another method 700 for generating output using an MLLM with optimized lexical / embedding is shown. Method 700 can be implemented by a computing device with suitable processor-executable instructions for causing the computing device to perform the operation. Method 700 can be implemented wholly or partially by computer system 110, computer system 400, or MLLM system 430 (see [link to documentation]). Figure 1 , Figure 4A and Figure 4B This can be implemented using computer systems such as computing devices (e.g., computing device 130) or associated clients. As part of method 700, the computer system can maintain the MLLM, and computing devices (e.g., computing device 130) or associated clients can access or use the MLLM by communicating with the computer system.

[0126] Method 700 begins with operation 702. At operation 702, the system receives at least one parameter. This at least one parameter can be received from the computing device. For example, the computing device can provide this at least one parameter to customize the performance of the MLLM. For example, the at least one parameter may include a budget (e.g., budget...). Alternatively, it may include tools that can be used to determine the budget. (For example, pruning ratio) data. Additionally or alternatively, this at least one parameter may include cost data. The system may define the mapping based at least in part on this at least one parameter. For example, the computing device can provide the system with... , and / or The expected value. Alternatively, the computing device can provide the system with information for determining... , and / or The guide to the value. For example, the guide could be a condition, such as... ,or and The ratio. It should also be noted that this at least one parameter can be received through user selection. For example, the system can cause the computing device to display optional options for the budget or cost of the MLLM. The computing device or associated client can select one of these options to provide the system with this at least one parameter.

[0127] Following operation 702, method 700 can proceed to operation 704. At operation 704, as previously discussed relative to operation 702, the system can determine the budget based on at least one parameter.

[0128] After operation 704, method 700 can proceed to operation 710. At operation 710, the system can receive query and associated multimodal input data. This system can be similar to the one referenced herein. Figure 5 The described method 500 performs operation 510 to execute operation 710.

[0129] Following operation 710, method 700 can proceed to operation 720. At operation 720, the system can encode the multimodal input data to generate multiple embeddings that map to a shared embedding space. This system can be similar to the one referenced herein. Figure 5 The described method 500 performs operation 520 to execute operation 720.

[0130] Following operation 720, method 700 can proceed to operation 730. At operation 730, the system can encode the query into the shared embedding space as a query embedding. This system can be similar to the one referenced herein. Figure 5 The described method 500 performs operation 530 to execute operation 730.

[0131] After operation 730, method 700 can proceed to operation 740. At operation 740, the system can select a subset of the multiple embeddings. This system can be similar to the one referenced in this paper. Figure 5 The described method 500 performs operation 540 to execute operation 740.

[0132] Following operation 740, method 700 can proceed to operation 750. At operation 750, the system can provide the selected subset of the embedding and the query embedding to the neural network of the MLLM to generate a response to the query. This system can be similar to the one described in this paper. Figure 5 The described method 500 performs operation 550 to execute operation 750.

[0133] In some implementations, the methods disclosed herein can be implemented as computer-executable instructions stored in one or more non-transitory computer-readable storage devices (in the form of software, firmware, or a combination thereof), such that when executed, these instructions cause one or more physical components (e.g., one or more circuits) to perform the methods disclosed herein.

[0134] For example, in some implementations, the methods disclosed herein can be performed using means including one or more processors functionally connected to one or more non-transitory computer-readable storage devices or media, the one or more non-transitory computer-readable storage devices or media storing computer-executable instructions of the methods disclosed herein, the one or more processors reading the computer-executable instructions from the one or more non-transitory computer-readable storage devices or media and executing the instructions to perform the methods disclosed herein.

[0135] In some implementations, the apparatus may not have any processor or computer-readable storage device or medium. Instead, the apparatus may include any other suitable physical or virtual components for implementing the methods disclosed herein.

[0136] In some implementations, the computer-executable instructions that implement the methods disclosed herein may be one or more computer programs, one or more program products, or a combination thereof.

[0137] In some implementations, the methods disclosed herein can be implemented as one or more circuits, one or more components, one or more units, one or more modules, one or more integrated-circuit (IC) chips, one or more chipsets, one or more devices, one or more apparatuses, one or more systems, etc.

[0138] One or more circuits, components, units, modules, IC chips, chipsets, devices, apparatuses, or systems can be physical, virtual, or a combination thereof. Here, the term “virtual” (e.g., “virtual device”) refers to circuits, components, units, modules, chipsets, devices, apparatuses, systems, etc., which are simulated or otherwise formed using suitable software or firmware to appear as “real” or physical.

[0139] This invention includes various embodiments, not only method embodiments, but also apparatus embodiments and other embodiments related to non-transitory computer-readable storage media. Embodiments may be combined, alone or in combination, with the features disclosed herein.

[0140] While this invention has been described with reference to illustrative embodiments, it is not intended to be construed in a limiting sense. Various modifications and combinations of the illustrative embodiments, as well as other embodiments of the invention, will be apparent to those skilled in the art upon reference to this specification.

[0141] Additionally or alternatively, features disclosed herein in the context of any particular embodiment may be implemented in other embodiments. For example, method embodiments may be implemented in apparatus, system, and / or computer program product embodiments. Furthermore, while embodiments are described primarily in the context of methods and apparatus, other implementations are contemplated as instructions stored in non-transitory computer-readable media, etc. Such media may store programs or instructions to perform any of the methods consistent with the present invention.

[0142] Those skilled in the art will understand that the above embodiments and / or their features can be customized, separated, and / or combined as needed or desired. Furthermore, while embodiments have been described above with reference to the accompanying drawings, those skilled in the art will understand that variations and modifications can be made without departing from the scope defined by the appended claims.

Claims

1. A computer-implemented method for generating output using a multimodal model, characterized in that, The method includes: Receive queries and associated multimodal input data, wherein the multimodal input data includes at least two modalities; For each modality of the multimodal input data, the modality of the input data is encoded using the corresponding encoder of the modality to generate multiple embeddings, which are mapped to a shared embedding space containing embeddings for all the modalities of the input data; The query is encoded into the shared embedding space as a query embedding; A subset of the embeddings in the shared embedding space is selected based on the following: The distance measurement result between the query embedding and the corresponding embedding in the shared embedding space; Map each embedding in the shared embedding space to a corresponding cost, the corresponding cost being based on the modality of the embedding; The selected subset of the embeddings and the query embeddings are provided to the neural network in the multimodal model to generate a response to the query.

2. The computer-implemented method according to claim 1, characterized in that, The distance measurement result is based on at least one of cosine similarity and L2 distance.

3. The computer-implemented method according to claim 1, characterized in that, The method further includes: Receives at least one parameter; The mapping is defined at least in part based on the at least one parameter.

4. The computer-implemented method according to claim 1, characterized in that, The corresponding cost of the embedding is defined by the hyperparameters of the multimodal model.

5. The computer-implemented method according to claim 1, characterized in that, The corresponding cost of the embedding is defined based on the computational cost associated with the neural network processing the embedding.

6. The computer-implemented method according to claim 5, characterized in that, The computational cost is based on at least one of memory usage, runtime, and processing power.

7. The computer-implemented method according to claim 1, characterized in that, Select the subset as follows: Maximize a value function, which at least measures the relevance of the subset to the query embedding; The total cost will be kept within budget, and the total cost will be obtained through the following means: The range is obtained by applying the mapping to the subset; Calculate at least one of the sum and average for the range.

8. The computer-implemented method according to claim 6, characterized in that, The method further includes: Before selecting the subset, at least one parameter is received by user selection; The budget is determined based on at least one of the parameters.

9. The computer-implemented method according to claim 6, characterized in that, The value function includes a covering function that measures the relevance of the subset relative to the query embedding, and the covering function measures the relevance in at least the following ways: For the first embedding in the subset, determine the first distance measurement result between the first embedding and the query embedding; For the second embedding in the subset, determine the second distance measurement result between the second embedding and the query embedding; A calculation operation is applied to the first distance measurement result and the second distance measurement result, wherein the calculation operation is one of a summation operation and an averaging operation.

10. The computer-implemented method according to claim 6, characterized in that, The value function includes a reward function that biases the value function toward the diversity of embeddings, the reward function applying the bias at least based on an internal distance measurement between a first embedding and a second embedding in the subset.

11. The computer-implemented method according to claim 6, characterized in that, Selecting the subset also includes: Obtain copies of the multiple embedded elements; Identify the marginal embeddings in the replicas, which maximize the marginal gain per unit cost in the value function, the unit cost being calculated by applying the mapping to the marginal embeddings; Add the marginal embedding to the selected embedding set; Remove the marginal embedding from the copy.

12. The computer-implemented method according to claim 1, characterized in that, The at least two modalities include at least one of text data, audio data, and visual data.

13. A computer system for generating output using a multimodal model, characterized in that, The computer system includes: processor; The memory stores instructions that, when executed by the processor, cause the processor to: Receive queries and associated multimodal input data, wherein the multimodal input data includes at least two modalities; For each modality of the multimodal input data, the modality of the input data is encoded using the corresponding encoder of the modality to generate multiple embeddings, which are mapped to a shared embedding space containing embeddings for all the modalities of the input data; The query is encoded into the shared embedding space as a query embedding; A subset of the embeddings in the shared embedding space is selected based on the following: The distance measurement result between the query embedding and the corresponding embedding in the shared embedding space; Map each embedding in the shared embedding space to a corresponding cost, the corresponding cost being based on the modality of the embedding; The selected subset of the embeddings and the query embeddings are provided to the neural network in the multimodal model to generate a response to the query.

14. The computer system according to claim 13, characterized in that, The distance measurement result is based on at least one of cosine similarity and L2 distance.

15. The computer system according to claim 13, characterized in that, The corresponding cost of the embedding is defined based on the computational cost associated with the neural network processing the embedding.

16. The computer system according to claim 13, characterized in that, Select the subset as follows: Maximize a value function, which at least measures the relevance of the subset to the query embedding; The total cost will be kept within budget, and the total cost will be obtained through the following means: The range is obtained by applying the mapping to the subset; Calculate at least one of the sum and average for the range.

17. The computer system according to claim 15, characterized in that, The value function includes a covering function that measures the relevance of the subset relative to the query embedding, and the covering function measures the relevance in at least the following ways: For the first embedding in the subset, determine the first distance measurement result between the first embedding and the query embedding; For the second embedding in the subset, determine the second distance measurement result between the second embedding and the query embedding; A calculation operation is applied to the first distance measurement result and the second distance measurement result, wherein the calculation operation is one of a summation operation and an averaging operation.

18. The computer system according to claim 15, characterized in that, The value function includes a reward function that biases the value function toward the diversity of embeddings, the reward function applying the bias at least based on an internal distance measurement between a first embedding and a second embedding in the subset.

19. The computer system according to claim 15, characterized in that, Selecting the subset also includes: Obtain copies of the multiple embedded elements; Identify the marginal embeddings in the replicas, which maximize the marginal gain per unit cost in the value function, the unit cost being calculated by applying the mapping to the marginal embeddings; Add the marginal embedding to the selected embedding set; Remove the marginal embedding from the copy.

20. A non-transitory computer-readable medium storing processor-executable instructions, characterized in that, When the processor-executable instructions are executed by the processor, the processor: Receive queries and associated multimodal input data, wherein the multimodal input data includes at least two modalities; For each modality of the multimodal input data, the modality of the input data is encoded using the corresponding encoder of the modality to generate multiple embeddings, which are mapped to a shared embedding space containing embeddings for all the modalities of the input data; The query is encoded into the shared embedding space as a query embedding; A subset of the embeddings in the shared embedding space is selected based on the following: The distance measurement result between the query embedding and the corresponding embedding in the shared embedding space; Map each embedding in the shared embedding space to a corresponding cost, the corresponding cost being based on the modality of the embedding; The selected subset of the embeddings and the query embeddings are provided to the neural network in the multimodal model to generate a response to the query.

Images