System and method for prompting search

Abstract

Systems and methods for prompting a search. A computer-implemented method includes: receiving a plurality of input images; generating a visual matrix using the plurality of images and an image encoder, wherein the visual matrix comprises a list of encoded images; receiving a plurality of text prompts; selecting a text prompt from the plurality of text prompts; sending a first one of the text cues to a language model to generate a list of candidate lexical elements; lexical elements are selected; converting the textual cue into an updated textual cue via the additional lemma; generating a text matrix by using a text prompt and a text encoder; and determining a score associated with the image-text similarity matrix using the values assigned at the image-text similarity matrix; and after evaluating each of the plurality of textual cues, in response to identifying the highest score associated with the final lexical element, evaluating a criterion and outputting the final lexical element to the updated textual cue.

Description

Technical Field

[0001] The present disclosure relates to hint engineering, including hint engineering utilizing language models, such as large language models (LLMs). Background Art

[0002] Self-supervised visual language models (VLMs), such as contrastive language image pre-training (CLIP), can create an aligned image encoder and text encoder via contrastive training. Unlike traditionally trained classification networks, such alignment enables zero-shot image classification by prompting the text encoder with a hand-crafted input such as “photo of {}” and then predicting the target via the maximum inner product of the embedding input image. However, selecting effective prompts for zero-shot learning is still largely an ad-hoc process: some systems have added several prompts such as “cartoon {}” or “art of {}”, which aim to improve performance and accuracy on ImageNet-R. Summary of the invention

[0003] The first illustrative embodiment describes a computer-implemented method for adjusting a pre-trained machine learning network. The method includes: receiving a plurality of input images including a plurality of pixels; generating a visual matrix using the plurality of input images and an image encoder of the machine learning network, wherein the visual matrix includes a list of encoded images; receiving a plurality of text prompts; selecting a first one of the text prompts from the plurality of text prompts; sending the first one of the text prompts to a large language model (LLM) to generate a candidate token list, wherein the candidate token list is generated by selecting a subset from each token associated with the first one of the text prompts, wherein the subset includes the most likely token associated with the first one of the text prompts, wherein the most likely token is calculated in response to an output of the LLM; selecting one or more tokens from the candidate list; converting one of the text prompts into an updated text prompt by appending the one or more selected tokens associated with the plurality of text prompts; generating a text matrix using the updated text prompt including the one or more tokens and a text encoder of the machine learning network, wherein the text matrix A method of detecting an image-text similarity matrix comprising a list of encoded visual descriptors, the list comprising an updated text prompt having one or more word-grams; multiplying a text matrix and a visual matrix to generate an image-text similarity matrix, wherein the image-text similarity matrix assigns numerical values ​​indicating similarity between each encoded visual descriptor and each encoded image, wherein the similarity is indicated by an entry of the image-text similarity matrix having the numerical value; determining a score associated with the image-text similarity matrix using the numerical values ​​assigned at the image-text similarity matrix; and when the score drops below a threshold, repeating steps (vi-xi) for a second word-gram of a first one of the text prompts, and when the score exceeds the threshold, adding one or more word-grams to the updated text prompt, and repeating steps (iv-xi) for the remainder of each of the plurality of text prompts; and after evaluating each of the plurality of text prompts, in response to identifying the highest score associated with the final word-gram, outputting a final word-gram to the updated text prompt.

[0004] A second illustrative embodiment describes a system comprising a processor programmed to: receive a plurality of input images; generate a visual matrix using an image encoder of the plurality of input images and a machine learning network, wherein the visual matrix includes a list of encoded images; receive a plurality of text prompts; select a first one of the text prompts from the plurality of text prompts; send the first one of the text prompts to a large language model (LLM) to generate a list of candidate word-grams, wherein the list of candidate word-grams is generated by selecting a subset of word-grams from each word-gram associated with the first one of the text prompts, wherein the subset includes a most probable word-gram associated with the first one of the text prompts, wherein the most probable word-gram is calculated in response to an output of the LLM; select one or more word-grams from the candidate list; convert one of the text prompts into an updated text prompt by appending the one or more selected word-grams associated with the plurality of text prompts; generate the updated text prompt using the text encoder of the machine learning network and the updated text prompt including the one or more word-grams; The invention relates to a method for generating an image-text similarity matrix comprising an image-text similarity matrix, wherein the image-text similarity matrix assigns a numerical value indicating a similarity between each encoded visual descriptor and each encoded image, wherein the similarity is indicated by an entry of the image-text similarity matrix having a numerical value; determining a score associated with the image-text similarity matrix using the numerical values ​​assigned at the image-text similarity matrix; repeating certain steps for a second word-gram of a first one of the text prompts when the score drops below a threshold, and adding one or more word-grams to the updated text prompt when the score exceeds the threshold, and repeating certain steps for the remainder of each of the plurality of text prompts; and outputting a final word-gram to the updated text prompt in response to identifying a highest score associated with the final word-gram after evaluating each of the plurality of text prompts.

[0005] A third illustrative embodiment describes a computer-implemented method comprising: receiving a plurality of input images; generating a visual matrix using an image encoder of the plurality of input images and a machine learning network, wherein the visual matrix comprises a list of encoded images; receiving a plurality of text prompts; selecting a first one of the text prompts from the plurality of text prompts; sending the first one of the text prompts to a language model (LM) to generate a list of candidate word-grams, wherein the list of candidate word-grams is a subset of less than all word-grams associated with the first one of the text prompts; selecting one or more word-grams from the candidate list; converting one of the text prompts into an updated text prompt by appending the one or more selected word-grams associated with the plurality of text prompts; generating a text matrix using a text encoder of the updated text prompt comprising the one or more word-grams and the machine learning network. , wherein the text matrix includes a list of encoded visual descriptors, the list including an updated text prompt having one or more word-grams; multiplying the text matrix and the visual matrix to generate an image-text similarity matrix, wherein the image-text similarity matrix assigns numerical values ​​indicating similarity between each encoded visual descriptor and each encoded image, wherein the similarity is indicated by an entry of the image-text similarity matrix having the numerical value; determining a score associated with the image-text similarity matrix using the numerical values ​​assigned in the image-text similarity matrix; and when the score drops below a threshold, repeating certain steps for a second word-gram of a first one of the text prompts, and when the score exceeds the threshold, adding one or more word-grams to the updated text prompt and repeating certain steps for the remainder of each of the plurality of text prompts. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 A system for training a neural network according to one embodiment is shown.

[0007] Figure 2 A computer-implemented method for training and utilizing a neural network is shown according to one embodiment.

[0008] Figure 3 A block diagram of an embodiment of a machine learning network according to searching with hints is illustrated.

[0009] Figure 4 A flow diagram of an embodiment of searching using prompts is illustrated.

[0010] Figure 5 Depicted is a schematic diagram of the interaction between a computer controlled machine and a control system according to one embodiment.

[0011] Figure 6 Depicted is a diagram according to an embodiment Figure 5Schematic diagram of a control system configured to control a vehicle, which can be a partially autonomous vehicle, a fully autonomous vehicle, a partially autonomous robot, or a fully autonomous robot.

[0012] Figure 7 Depicted Figure 5 Schematic diagram of a control system configured to control a manufacturing machine (such as a punch cutter, a cutting machine, or a gun drill) of a manufacturing system (such as part of a production line).

[0013] Figure 8 Depicted Figure 5 Schematic diagram of a control system configured to control a power tool (such as a drill or driver) having at least a partially autonomous mode.

[0014] Fig. 9 Depicted Figure 5 Schematic diagram of a control system configured to control an automated personal assistant.

[0015] Fig.10 Depicted Figure 5 Schematic diagram of a control system configured to control a monitoring system (such as a controlled access system or a surveillance system).

[0016] Fig.11 Depicted Figure 5 Schematic diagram of a control system configured to control an imaging system (eg, an MRI device, an x-ray imaging device, or an ultrasound device). DETAILED DESCRIPTION

[0017] Embodiments of the present disclosure are described. Embodiments of the present disclosure are described herein. However, it should be understood that the disclosed embodiments are merely examples, and other embodiments may take various and alternative forms. The figures are not necessarily to scale; some features may be enlarged or minimized to show the details of a particular component. Therefore, the specific structural and functional details disclosed herein should not be interpreted as restrictive, but merely as a representative basis for teaching those skilled in the art to adopt the embodiments in various ways. As will be understood by those of ordinary skill in the art, the various features illustrated and described with reference to any one of the figures may be combined with the features illustrated in one or more other figures to produce embodiments that are not explicitly illustrated or described. The combination of the illustrated features provides representative embodiments for typical applications. However, for specific applications or embodiments, various combinations and modifications of features consistent with the teachings of the present disclosure may be desirable.

[0018] As used herein, "a", "an", and "the" refer to both singular and plural referents unless the context clearly dictates otherwise. As an example, a "processor" programmed to perform various functions refers to one processor programmed to perform each function, or more than one processor that are collectively programmed to perform each of the various functions.

[0019] Generalization bounds provide statistical guarantees on the mean-case performance of the output of a learning algorithm. Traditional methods for deriving generalization bounds are inadequate to explain the generalization power of deep learning, which has spawned a range of new methods for deriving tighter generalization bounds for deep neural networks. In the recent literature on generalization bounds for neural networks, a large focus has been on developing data-dependent bounds, or bounds that consider the distribution of the data in addition to the hypothesis class. One of the best such bounds is based on the PAC-Bayes framework and is derived by constraining the KL divergence between well-informed priors and posteriors produced by the learning algorithm. PAC-Bayes bounds have led to the first non-empty generalization bounds for deep learning, but they are still too loose to be practically useful. In fact, many novel PAC-Bayes bounds with data-dependent priors, while non-empty, can be best described as validation bounds - for example, the use of data-dependent priors effectively exploits the held-out data in a manner more akin to cross-validation, which renders them not directly comparable to pure training performance bounds. Fundamentally, it may simply be the case that the use of isotropic Gaussians on large parameter spaces has fundamental limitations on reasoning about generalization in deep learning. Nevertheless, despite the lack of a clear theoretical basis, modern machine learning models are becoming larger and larger. In fact, large pre-trained models are becoming the main body in the design of modern machine learning models. One example is to use these pre-trained models (such as CLIP or ALIGN) as feature extractors and provide weak supervision for downstream target tasks via prompts, which are text descriptions of the desired tasks, which are usually significantly easier than the complete model weights or even the general linear classifier on the last layer. The versatility and performance of the models pre-trained by prompts have led to the rise of prompt engineering, an emerging paradigm in machine learning, in which users carefully design task specifications in text, or even learn prompts in a data-driven manner. Despite its empirical success, little is known about how and why these pre-trained models of prompts work, and in particular why the method seems to rarely suffer from overfitting: manually adjusting or even greedily optimizing prompts on a given training set usually performs almost as well on the corresponding test set.

[0020] In one embodiment, the systems and methods can show that traditional PAC-Bayes bounds are often very tight when applied to discrete hypothesis classes defined by prompts (and specifically using priors given by large language models), even for large domains: for example, for a full ImageNet classifier, a generalization bound of 31% error can be achieved, which is within 6% of the actual observed test error. This represents a huge improvement over traditional deep learning-based bounds, where achieving any non-empty bound on a domain like ImageNet often requires a lot of effort; see, for example, Table 1 comparing our bounds to other methods, especially variants that do not use data-dependent priors (as our prompt-based bounds do not). In summary, the system can find that, unlike traditional deep learning models, models pre-trained with prompts do not suffer from problems around generalization bounds, and one can easily derive strong theoretical guarantees for the use of prompts via well-studied techniques. Overall, these findings suggest that prompt engineering may be a principled approach for using these pre-trained models, despite a large amount of automatic or manual tuning, that does not suffer from the same lack of theoretical foundation as traditional deep learning models.

[0021] In the present disclosure, systems and methods can demonstrate that fairly simple analytical tools actually capture this behavior very well. In particular, the system can show that traditional PAC-Bayes bounds are often very tight when applied to discrete hypothesis classes defined by prompts (and specifically using priors given by large language models), even for large domains: for example, the system can achieve a generalization bound of 31% error for the full ImageNet classifier, which is within 6% of the actual observed test error. This represents a huge improvement over traditional deep learning-based bounds, in which achieving any non-empty bound on a domain like ImageNet often requires a lot of effort; see, for example, Table 1 comparing our bounds to other methods, especially variants that do not use data-dependent priors (as our prompt-based bounds do not). In summary, unlike traditional deep learning models, models pre-trained with prompts do not suffer from problems around generalization bounds, and one can easily derive strong theoretical guarantees for using prompts via well-studied techniques. Overall, these findings suggest that prompt engineering may be a principled approach for using these pre-trained models that do not suffer from the same lack of theoretical foundation as traditional deep learning models, despite extensive automatic or manual tuning.

[0022] With the advent of large pre-trained models, prompts were developed as a surprising but effective way to exploit the power of these large models with limited labeled data. The flexibility of prompts has enabled a wide range of new capabilities not available to previous machine learning models, leading to a significant effort to record successful prompting methods in both classification and text-to-image generation. Since many of these methods may appear to be heuristic or over-engineered, methods have been proposed to learn "optimal" prompts given labeled data, which perform well empirically and are parameter efficient. One disadvantage of such data-driven methods is that they learn "soft" prompts or embedding vectors that are not interpretable, and therefore do not exhibit good generalization properties. As such, existing methods have proposed gradient-based methods to learn interpretable prompts.

[0023] Cue engineering has been extended to computer vision through CLIP (Contrastive Language-Image Pre-training). CLIP uses a neural network architecture that combines natural language processing and computer vision to understand images and their corresponding text descriptions. It is pre-trained using a large dataset of images and text with a contrastive learning objective, allowing the model to distinguish between a wide variety of image and text combinations.

[0024] This extensive training allows CLIP to grasp various concepts and relationships between images and language. Unlike traditional computer vision models that rely on fixed labels for image classification, CLIP can perform multiple tasks based on natural language instructions. Examples include object recognition, image caption generation, and zero-shot image classification using text descriptions, even for unseen labels.

[0025] A generalization bound can be an upper bound on the generalization gap of a model. It has always been difficult to derive such bounds for deep learning, and most are usually empty. They may also suffer from fundamental limitations. The core component of a generalization bound is a complexity measure, a quantity related to some aspect of generalization. As long as the complexity measure does not have access to the validation set, it can depend on the nature of the trained model, the optimizer, and possibly the training data. The most classical bounds such as VCdimension are usually related to some form of parameter count, which is usually too pessimistic for deep neural networks. Norm-based bounds usually rely on the margin of the model weights and some norm, but these bounds have been invalid when studying generalization in deep learning. Another major category is the PAC-Bayes bound, which has been much more successful in deep learning due to the flexibility of the prior, although these bounds are still much looser than the actual generalization error. In one approach, a system can belong to the PAC-Bayes family, but the system can apply the PAC-Bayes bound to the distribution of discrete tokens instead of the parameters of the neural network with the language model as a prior. This can allow the system to derive significantly tighter bounds compared to applying a PAC-Bayes bound that uses less informative priors.

[0026] Reference is now made to the embodiments illustrated in the figures, which may apply these teachings to a machine learning model or neural network. Figure 1 A system 100 for training a neural network (e.g., a deep neural network) is shown. The system 100 may include an input interface for accessing training data 102 for the neural network. For example, Figure 1 As illustrated in , the input interface may be constituted by a data storage interface 104 that can access the training data 102 from a data storage device 106. For example, the data storage interface 104 may be a memory interface or a persistent storage interface, such as a hard disk or SSD interface, but may also be a private, local or wide area network interface, such as a Bluetooth, Zigbee or Wi-Fi interface or an Ethernet or fiber optic interface. The data storage device 106 may be an internal data storage device of the system 100, such as a hard disk drive or SSD, but may also be an external data storage device, for example, a network accessible data storage device.

[0027] In some embodiments, the data storage device 106 may further include a data representation 108 of an untrained version of the neural network, which the system 100 can access from the data storage device 106. However, it will be appreciated that the training data 102 and the data representation 108 of the untrained neural network can also each be accessed from different data storage devices, for example, via different subsystems of the data storage interface 104. Each subsystem can have a data storage interface 104 of the type described above. In other embodiments, the data representation 108 of the untrained neural network can be generated internally by the system 100 based on the design parameters of the neural network, and thus may not be explicitly stored on the data storage device 106. The system 100 can further include a processor subsystem 110, which can be configured to provide an iterated function as a replacement for the stack of neural network layers to be trained during operation of the system 100. Here, the corresponding layers of the replaced layer stack can have mutually shared weights and can receive as input the output of the previous layer, or, for the first layer of the layer stack, receive as input the initial activation and a portion of the input of the layer stack. The processor subsystem 110 may be further configured to iteratively train the neural network using the training data 102. Here, the training iterations of the processor subsystem 110 may include a forward propagation portion and a backward propagation portion. The processor subsystem 110 may be configured to perform the forward propagation portion by: determining an equilibrium point of the iterative function at which the iterative function converges to a fixed point, in addition to defining other operations of the forward propagation portion that may be performed, wherein determining the equilibrium point includes using a numerical root finding algorithm to find a root solution of the iterative function minus its input; and by providing the equilibrium point as a substitute for the output of the stack of layers in the neural network. The system 100 may further include an output interface for outputting a data representation 112 of the trained neural network, which data may also be referred to as trained model data 112. For example, as also Figure 1 , the output interface may be comprised of a data storage interface 104, which in these embodiments is an input / output ("IO") interface, via which trained model data 112 may be stored in a data storage device 106. For example, a data representation 108 defining an "untrained" neural network may be replaced, at least in part, with a data representation 112 of a trained neural network during or after training, as parameters of the neural network (such as weights, hyperparameters, and other types of parameters of the neural network) may be adapted to reflect the training on the training data 102. Figure 1, this is also illustrated by reference numerals 108, 112 referring to the same data records on the data storage device 106. In other embodiments, the data representation 112 may be stored separately from the data representation 108 defining the "untrained" neural network. In some embodiments, the output interface may be separate from the data storage interface 104, but may generally be of the type described above for the data storage interface 104.

[0028] The structure of system 100 is one example of a system that can be used to train a pre-trained machine learning network that utilizes few-shot image learning as described herein. Figure 2 Additional structures for operating and training machine learning models are shown in .

[0029] Figure 2 A system implementing a machine learning model described herein (e.g., a pre-trained machine learning network using few-shot image learning described herein) is depicted. System 200 can be implemented to perform few-shot image learning as described herein. System 200 can include at least one computing system 202. Computing system 202 can include at least one processor 204, which is operatively connected to a memory unit 208. Processor 204 can include one or more integrated circuits that implement the functionality of a central processing unit (CPU) 206. CPU 206 can be a commercially available processing unit that implements an instruction set such as one of the x86, ARM, Power, or MIPS instruction set families. During operation, CPU 206 can execute stored program instructions retrieved from memory unit 208. The stored program instructions can include software that controls the operation of CPU 206 to perform the operations described herein. In some examples, processor 204 can be a system on chip (SoC) that integrates the functionality of CPU 206, memory unit 208, network interface, and input / output interface into a single integrated device. The processor may include a controller, a tensor processing unit, a graphics processing unit, an ASIC, an FPGA, etc. The computing system 202 may implement an operating system for managing various aspects of operation. Figure 2 One processor 204, one CPU 206, and one memory 208 are shown, but of course more than one of each may be utilized in the overall system.

[0030] The memory unit 208 may include volatile memory and non-volatile memory for storing instructions and data. Non-volatile memory may include solid-state memory, such as NAND flash memory, magnetic and optical storage media, or any other suitable data storage device that retains data when the computing system 202 is disabled or loses power. Volatile memory may include static and dynamic random access memory (RAM) that stores program instructions and data. For example, the memory unit 208 may store a machine learning model 210 or algorithm, a training data set 212 for the machine learning model 210, and an original source data set 216.

[0031] The computing system 202 may include a network interface device 222 that is configured to provide communications with external systems and devices. For example, the network interface device 222 may include a wired and / or wireless Ethernet interface as defined by the Institute of Electrical and Electronics Engineers (IEEE) 802.11 series of standards. The network interface device 222 may include a cellular communication interface for communicating with a cellular network (e.g., 3G, 4G, 5G). The network interface device 222 may further be configured to provide a communication interface to an external network 224 or a cloud.

[0032] The external network 224 may be referred to as the World Wide Web or the Internet. The external network 224 may establish a standard communication protocol between computing devices. The external network 224 may allow information and data to be easily exchanged between computing devices and the network. One or more servers 230 may communicate with the external network 224.

[0033] The computing system 202 may include an input / output (I / O) interface 220, which may be configured to provide digital and / or analog input and output. The I / O interface 220 is used to transmit information between an internal storage device and an external input and / or output device (e.g., an HMI device). The I / O interface 220 may include an associated circuit or bus (BUS) network to transmit information to (one or more) processors and storage devices, or to transmit information between (one or more) processors and storage devices. For example, the I / O interface 220 may include a digital I / O logic line that can be read or set by (one or more) processors, a handshake line for supervising data transmission via the I / O line, a timing and counting facility, and other known structures for providing such functions. Examples of input devices include keyboards, mice, sensors, etc. Examples of output devices include monitors, printers, speakers, etc. The I / O interface 220 may include an additional serial interface (e.g., a universal serial bus (USB) interface) for communicating with external devices.

[0034] The computing system 202 may include a human-machine interface (HMI) device 218, which may include any device that enables the system to receive control inputs. Examples of input devices may include human-machine interface inputs, such as keyboards, mice, touch screens, voice input devices, and other similar devices. The computing system 202 may include a display device 232. The computing system 202 may include hardware and software for outputting graphics and text information to the display device 232. The display device 232 may include an electronic display screen, a projector, a printer, or other suitable devices for displaying information to a user or operator. The computing system 202 may be further configured to allow interaction with a remote HMI and a remote display device via a network interface device 222.

[0035] The system can be implemented using one or more computing systems. Although this example depicts a single computing system 202 that implements all described features, it is intended that various features and functions be separated and implemented by multiple computing units that communicate with each other. The specific system architecture selected may depend on various factors.

[0036] The system can implement a machine learning algorithm 210 configured to analyze a raw source data set 216. The raw source data set 216 can include raw or unprocessed sensor data that can represent an input data set for the machine learning system. The raw source data set 216 can include video, video segments, images, text-based information, audio or human speech, time series data (e.g., pressure sensor signals over time), and raw or partially processed sensor data (e.g., a radar map of an object). Figure 5-Figure 11 Several different examples of inputs are shown and described. In some examples, the machine learning algorithm 210 can be a neural network algorithm (e.g., a deep neural network) designed to perform a predetermined function. For example, a neural network algorithm can be configured in an automotive application to identify street signs or pedestrians in an image. The machine learning algorithm(s) 210 may include an algorithm configured to operate a pre-trained machine learning network that utilizes few-shot image learning as described herein.

[0037] The computer system may store a training data set 212 for the machine learning algorithm 210. The training data set 212 may represent a previously constructed data set for training the machine learning algorithm 210. The machine learning algorithm 210 may use the training data set 212 to learn weighting factors associated with the neural network algorithm. The training data set 212 may include a source data set having a corresponding achievement or result that the machine learning algorithm 210 attempts to replicate via the learning process. In this example, the training data set 212 may include an input image containing an object (e.g., a street sign). The input image may include various scenes in which the object is identified.

[0038] The machine learning algorithm 210 can operate in a learning mode using the training data set 212 as input. The machine learning algorithm 210 can be executed in multiple iterations using data from the training data set 212. For each iteration, the machine learning algorithm 210 can update the internal weighting factors based on the achieved results. For example, the machine learning algorithm 210 can compare the output results (e.g., the reconstructed or supplemented image in the case where the image data is the input) with those included in the training data set 212. Since the training data set 212 includes the expected results, the machine learning algorithm 210 can determine when the performance is acceptable. After the machine learning algorithm 210 reaches a predetermined performance level (e.g., 100% consistent with the results associated with the training data set 212) or converges, the machine learning algorithm 210 can be executed using data that is not in the training data set 212. It should be understood that in the present disclosure, "convergence" can mean that a set (e.g., predetermined) number of iterations have occurred, or the residual is small enough (e.g., the change in the approximation probability within the iteration changes less than a threshold), or other convergence conditions. The trained machine learning algorithm 210 can be applied to a new data set to generate annotated data.

[0039] The machine learning algorithm 210 may be configured to identify specific features in the raw source data 216. The raw source data 216 may include multiple instances or input data sets for which supplementary results are desired. For example, the machine learning algorithm 210 may be configured to identify the presence of road signs in a video image and annotate the occurrence. The machine learning algorithm 210 may be programmed to process the raw source data 216 to identify the presence of specific features. The machine learning algorithm 210 may be configured to identify the features in the raw source data 216 as predetermined features (e.g., road signs). The raw source data 216 may be derived from various sources. For example, the raw source data 216 may be actual input data collected by the machine learning system. The raw source data 216 may be machine generated in order to test the system. As an example, the raw source data 216 may include raw video images from a camera.

[0040] In one example, the original source data 216 may include image data representing an image. Applying the machine learning algorithms described herein (eg, few-shot image learning, CLIP model, etc.), the output may be a tuned network associated with a collection of images.

[0041] Figure 3 An overview of the diagram including the prompting project is shown. is the input set and y = [K] is the label set, and there is an unknown probability distribution D on (X×Y). Suppose our data (X1, Y1), ..., (X η , Y η ) are random variables drawn independently and identically from D, and consider a predictor f:X→Y and a fixed set of predictors indexed by a parameter set Θ. This description can be used with f θ to denote the classifier indexed by θ. The description can be considered as The generalization error (risk) of the predictor can be defined as And for the sample S = [(X1, Y1), ..., (X n , Y n )], empirical risk satisfy An estimator is a function

[0042] CLIP consists of two encoders f img and f txt One or more images 301 may be sent to an image encoder 303 as input to an image encoder. Text hints 302 may be utilized in a text encoder 311. Given an image Image Encoder Maps image X to a d-dimensional real-valued embedding. Given a text Image Encoder Map the sentence Z to a d-dimensional real-valued embedding. The text matrix 313 can be output by the image encoder. Given a batch of images and their corresponding texts The training objective is to maximize the cosine similarity of the embeddings of matching image and text pairs, and minimize the cosine similarity of image and text pairs that do not correspond to each other. The main task to be considered is to use natural language cues via a pre-trained visual language model by finding a category cue θ for each category. k to classify the image. For a k-class classification problem, the zero-shot classifier using CLIP is:

[0043]

[0044] As discussed, the system is typically interested in the generalization ability of the predictor, and the system may quantify this by an upper bound on the population loss (true risk). In general, the system may focus on the generalization gap, which is the difference between the true risk and the empirical risk. Various generalization bounds may depend on the implicit bias of the learning algorithm, the training data S, or the data generating distribution D. The classical VC bound may not depend on any of them. Distribution-dependent bounds are expressed in quantities related to the data generating distribution, while data-dependent bounds are expressed in empirical quantities that can be computed directly from the data and can be used for self-certification.

[0045] The PAC-Bayes framework defines a hierarchy of hypotheses in our hypothesis class Θ, which takes the form of a prior distribution P over Θ. That is, the system can assign a probability P(θ) ≥ 0 to each θ∈Θ, and call P(θ) the prior score of θ. The learning process defines a posterior probability over Θ, which can be denoted by Q. In the context of supervised learning, the system can think of Q as defining the following prediction rule: given an instance X, the system can randomly pick a hypothesis θ according to Q and predict f θ (X). Notably, it is shown that the generalization gap can be upper bounded by the KL divergence between P and Q.

[0046] In one embodiment, the hint can be similar to finding a set of weights in a regular machine learning model, where the hypothesis space is the space of texts / words. The goal is to find a hint that maximizes the training accuracy without fine-tuning the model parameters. This process can be formulated as a discrete optimization over the word-word space V. Assuming that the system may be looking for a category hint 302 of length L, then the system will find the correct answer in |V| K·L The system searches for K·L tokens in the space. This search is exponential in the length of prompt 302, which can be intractable for even a small token space. To avoid this problem, the system can generate prompts 302 in a sequential manner; that is, the system can generate prompts 302 by selecting from a set of candidate tokens on the training dataset. The word-grams that maximize the rch criterion J are selected to add hints. The search criterion is the goal to be optimized, and the candidate word-grams are permissible word-grams that can be used to expand the current category hints. At each step of the search, for all but one category, the system can keep the category hints fixed. The hint for each category k is a sequence of l<L word-grams, And for θ k The next token of is obtained by:

[0047]

[0048] Label the category cues for all categories except the kth category. The pseudocode for this process is described in detail in Algorithm 1. Using || to indicate cascading, the simplest search instance is a greedy search, which uses:

[0049]

[0050] In other words, the system can always search all tokens in the inner loop to maximize the training accuracy. This simplest greedy search can be viewed as an instance of empirical risk minimization (ERM), since one of its only goals is to minimize the training error.

[0051] There may be drawbacks to this algorithm, one of which may be that the system may need to search V thoroughly in the inner loop. This may be expensive because it consists of all the tokens used by the visual language model (e.g., CLIP has about 50,000 tokens). Instead, the system may search only on a subset of V. To narrow this search space, the system may use a language model 305 (LM or LLM) to induce a search in the form of θ. k The distribution over the next token conditioned on is, and only tokens with high probability are evaluated:

[0052]

[0053] Since CLIP can be trained with natural language supervision, it is possible that a reasonable next word-gram can be captured by an LLM 305 (e.g., an autoregressive LM) that is trained to model the probability of the next word-gram. Then, at box 307, the system can take the top N candidates and evaluate the accuracy of these candidates. In one embodiment, the system can evaluate the accuracy of only the top N candidates. Conveniently, this can be viewed as constraining the complexity of the prompt because the language model 305 provides a structured prior on the set of word-grams. The system can observe that using the language model 305 to propose possible word-grams results in minimal performance loss, indicating that the language model is indeed good before searching for category prompts on the image classification task. In addition, the system can use predefined strings to further constrain the space of hypotheses by starting with an initial prompt such as "This is an image of..." instead of using an empty string. These provide additional structure to the generated prompts.

[0054] This process can be further extended to optimize the PAC-Bayes bound via structural risk minimization (SRM). That is, the system can take hypothesis complexity (e.g., KL divergence) into account as it searches for the next token for each hint.

[0055] Algorithm 1 can be shown below:

[0056] For

[0057] 1.ClassOrder←A randomly ordered set of categories

[0058] 2.For k∈ClassOrder:

[0059] a. Use the language model to calculate for each v in the vocabulary (That is, each token in the vocabulary follows the current textual cue of category k Probability of

[0060] b. ←According to p LM The top N most likely next words

[0061] c.criteria←-∞

[0062] d.For

[0063] i.score ← when v is appended to the current category k prompt The accuracy of CLIP predictions made at

[0064] ii. Optional: (plus the word v follows the text prompt The probability of multiplying by a constant β, which is a hyperparameter)

[0065] iii.If score>criteria: (then this is the best token seen so far in this loop, and the system should add it to the hint)

[0066] 1.criteria←score

[0067] 2.

[0068] Returnθ

[0069] Thus, when LLM 305 generates a list of N most similar word-grams at step 307, the system can then append one of the word-grams to the prompt at step 309. The updated prompt including the new word-gram can be sent to text encoder 311 to create a text matrix 315. An image-text similarity matrix 317 can be generated by multiplying the text matrix 315 with the image matrix 313. The image-text similarity matrix 317 can include a numerical value for determining a score that is used to evaluate the word-gram (e.g., an updated text prompt with the word-gram). At decision 319, the system can determine whether the score exceeds a certain criterion or threshold. If the score associated with the word-gram meets the criterion, or performs better than the previous word-gram, it can be added to the prompt 302. If not, then at step 321, the word-gram can be removed and the system can evaluate the new word-gram. The system can evaluate the set of all word-grams for the associated category of the prompt 302.

[0070] In principle, the system could use KL divergence directly in the target optimization, but using the probability of tokens as a proxy produces far more linguistically consistent hints without sacrificing the quality of the solution. Once again, the system can perform this optimization in a sequential manner via Greedy:

[0071]

[0072] where β is a hyperparameter that controls the strength of regularization, and TopN(·) is a The number of tokens to search is also a hyperparameter that can be adjusted based on computational constraints. This type of search may serve as a regularized greedy.

[0073] As mentioned earlier, deriving generalization bounds can be closely related to assigning a priori probabilities to hypotheses that they are good hypotheses (i.e., before seeing training data). One of the naive approaches might be to assign a uniform probability to each hypothesis. With a uniform prior, the system can have a known uniform convergence bound. For every δ>0, where probability is 1-δ on a training set of size n, for any posterior θ∈Θ, the following holds

[0074]

[0075] Since the space of all cues may be discrete, for a single hypothesis ˆθ, the system can have the following uniform convergence bounds for cues that depend on the cues length, number of categories, and number of tokens:

[0076]

[0077] However, not all cues are equally likely to be correct. To obtain tighter generalization guarantees on the learned Θ, the system can exploit the classic PAC-Bayes bound to derive an upper bound on the generalization error with respect to the learned cues. For every δ>0, a prior P on Θ with probability 1-δ on a training set of size n, for any posterior Q on Θ, the following holds:

[0078]

[0079] In traditional deep learning, P and Q are usually chosen to be isotropic Gaussian distributions, so the KL divergence between the prior and the posterior can be easily calculated. The system and method can use the language model as a prior on K independent prompts, In addition, the system can use tips found through search or through the Tips Project Treat as point In this case, the KL divergence is conveniently equal to The negative log-likelihood of It is zero everywhere except:

[0080]

[0081] This bound can provide an intuitive explanation that the generalization cue is the cue that leads to good training performance, and is possible under the language model. With a point-quality posteriori on the discrete space, the system is free to de-randomize the PAC-Bayes bound. Combining these observations, the following deterministic upper bound on the generalization error can be used:

[0082]

[0083] The system can refer to the RHS of the inequality as

[0084] Figure 4A flow chart according to an embodiment of the present disclosure is illustrated. A machine learning network may include a CLIP model having a text encoder and an image encoder. At step 401, the network may receive one or more text prompts. The text prompt may be associated with a category. At step 403, the model may select a text prompt to be utilized at the network. At step 403, the system may generate a list of word-grams associated with the selected text prompt. The word-gram list may be a list of words or word strings that may be associated with the category of the text prompt. At step 405, the system may send one of the text prompts to a language model, such as a large language model (LLM), to generate a list of candidate word-grams. The list of candidate word-grams may be generated by selecting a subset from each word-gram associated with the first one in the text prompt. Therefore, the subset will be smaller than each available word-gram. As discussed above, the subset may include the most likely word-gram associated with the first one in the text prompt. The most likely word-gram may be calculated using a language model or the output of a language model, such as an LLM. At step 407, the system may select one or more word-grams from the candidate list. As discussed above, word-grams can be selected randomly or via any method. At step 409, the system can update one of the text prompts or convert one of the text prompts into an updated text prompt via adding one or more selected word-grams associated with a plurality of text prompts. Therefore, for example, the word-gram of "door" can be added to the text prompt "the vehicle has ...". Utilizing the updated text prompt, the system can generate a text matrix at step 411. The system can generate the text matrix using both the updated text prompt and the text encoder of the network. The text matrix can include a list of coded visual descriptors, which includes updated text prompts with one or more word-grams. At step 413, the system can identify the score associated with the word-gram. The score can be calculated or determined based on the image-text similarity matrix created using the text matrix and the image matrix. The system can multiply the text matrix and the visual matrix to generate the image-text similarity matrix. The image-text similarity matrix can assign a numerical value indicating the similarity between each coded visual descriptor and each coded image. Similarity can be indicated by the entry of the image-text similarity matrix with a numerical value. At decision 415, the system can determine whether the score meets the criteria. If the score drops below a threshold, the system can continue to evaluate other word-grams until a sufficient score is met. When the score exceeds a threshold, the system can add one or more word-grams to the updated text prompt and continue to evaluate the remainder of each of the multiple text prompts. The criterion threshold can be a certain score (e.g., a threshold score, a score from a previous prompt and word-gram, etc.), a number of iterations, a convergence threshold (e.g., based on a loss function algorithm, etc.), etc. When adding word-grams to the updated text prompt, the system can continue to evaluate other text prompts associated with the prompt of the list by selecting other word-grams and comparing them.If a word-gram with a higher score is found, it can replace the previous text prompt (with the lower-scoring word-gram). When all word-grams have been evaluated, the system can output the final word-gram to the updated text prompt when the system identifies the highest score associated with the final word-gram after evaluating each of the multiple text prompts. The final word-gram utilized can include the best score associated with the text prompt among all word-grams. The system can run this evaluation until all text prompts of the candidate list have been evaluated.

[0085] Figure 5 A diagram of the interaction between a computer-controlled machine 500 and a control system 502 is depicted. The computer-controlled machine 500 includes an actuator 504 and a sensor 506. The actuator 504 may include one or more actuators, and the sensor 506 may include one or more sensors. The sensor 506 is configured to sense a condition of the computer-controlled machine 500. The sensor 506 may be configured to encode the sensed condition into a sensor signal 508 and transmit the sensor signal 508 to the control system 502. Non-limiting examples of the sensor 506 include video, radar, lidar, ultrasonic, and motion sensors. Image data such as video images, picture images, radar images, sound images, etc. may be retrieved from these sensors. The image may represent video or picture data, which may include a plurality of pixels forming a scene. A pixel may be the smallest addressable element in a raster image, or the smallest addressable element in a dot matrix display device. In most digital display devices, a pixel may be the smallest element that can be manipulated by software. Each pixel may be a sample of an original or synthesized image. In one embodiment, more samples generally provide a more accurate representation of the original. The intensity of each pixel may be variable. In one embodiment, sensor 506 is an optical sensor configured to sense an optical image of an environment proximate to computer-controlled machine 500.

[0086] The control system 502 is configured to receive sensor signals 508 from the computer controlled machine 500. As explained below, the control system 502 may be further configured to calculate actuator control commands 510 depending on the sensor signals and transmit the actuator control commands 510 to the actuators 504 of the computer controlled machine 500.

[0087] like Figure 5As shown in , the control system 502 includes a receiving unit 512. The receiving unit 512 can be configured to receive the sensor signal 508 from the sensor 506 and transform the sensor signal 508 into an input signal x. In an alternative embodiment, the sensor signal 508 is directly received as the input signal x without the receiving unit 512. Each input signal x can be a part of each sensor signal 508. The receiving unit 512 can be configured to process each sensor signal 508 to generate each input signal x. The input signal x can include data corresponding to the image recorded by the sensor 506.

[0088] The control system 502 includes a classifier 514. The classifier 514 can be configured to classify the input signal x into one or more labels using a machine (ML) learning algorithm, such as the neural network described above. The classifier 514 is configured to be parameterized by parameters, such as those described above (e.g., parameter θ). The parameter θ can be stored in and provided by a non-volatile storage device 516. The classifier 514 is configured to determine an output signal y from the input signal x. Each output signal y includes information that assigns one or more labels to each input signal x. The classifier 514 can transmit the output signal y to a conversion unit 518. The conversion unit 518 is configured to convert the output signal y into an actuator control command 510. The control system 502 is configured to transmit the actuator control command 510 to the actuator 504, which is configured to actuate the computer-controlled machine 500 in response to the actuator control command 510. In another embodiment, the actuator 504 is configured to actuate the computer-controlled machine 500 directly based on the output signal y.

[0089] When the actuator 504 receives the actuator control command 510, the actuator 504 is configured to perform an action corresponding to the associated actuator control command 510. The actuator 504 may include control logic configured to transform the actuator control command 510 into a second actuator control command for controlling the actuator 504. In one or more embodiments, the actuator control command 510 may be used to control a display instead of or in addition to an actuator.

[0090] In another embodiment, the control system 502 includes the sensor 506 instead of or in addition to the computer-controlled machine 500 including the sensor 506. The control system 502 may also include the actuator 504 instead of or in addition to the computer-controlled machine 500 including the actuator 504.

[0091] like Figure 5As shown in , the control system 502 also includes a processor 520 and a memory 522. The processor 520 may include one or more processors. The memory 522 may include one or more memory devices. The classifier 514 of one or more embodiments (e.g., a machine learning algorithm, such as those described above with respect to an image encoder or a text encoder) can be implemented by the control system 502, which includes a non-volatile storage device 516, a processor 520, and a memory 522.

[0092] The non-volatile storage 516 may include one or more persistent data storage devices, such as a hard drive, an optical drive, a tape drive, a non-volatile solid-state device, a cloud storage device, or any other device capable of persistently storing information. The processor 520 may include one or more devices selected from a high-performance computing (HPC) system, including a high-performance core, a microprocessor, a microcontroller, a digital signal processor, a microcomputer, a central processing unit, a field programmable gate array, a programmable logic device, a state machine, a logic circuit, an analog circuit, a digital circuit, or any other device that manipulates signals (analog or digital) based on computer executable instructions residing in the memory 522. The memory 522 may include a single memory device or multiple memory devices, including but not limited to random access memory (RAM), volatile memory, non-volatile memory, static random access memory (SRAM), dynamic random access memory (DRAM), flash memory, cache memory, or any other device capable of storing information.

[0093] The processor 520 may be configured to read into the memory 522 and execute computer executable instructions residing in the non-volatile storage 516 and embodying one or more ML algorithms and / or methods of one or more embodiments. The non-volatile storage 516 may include one or more operating systems and applications. The non-volatile storage 516 may store computer programs created using various programming languages ​​and / or technologies to compile and / or interpret computer programs, including but not limited to Java, C, C++, C#, Objective C, Fortran, Pascal, JavaScript, Python, Perl, and PL / SQL, alone or in combination.

[0094] When executed by the processor 520, the computer executable instructions of the non-volatile storage device 516 may cause the control system 502 to implement one or more of the ML algorithms and / or methods as disclosed herein. The non-volatile storage device 516 may also include ML data (including data parameters) that support the functions, features, and processes of one or more embodiments described herein.

[0095] The program code embodying the algorithms and / or methods described herein can be distributed individually or collectively as a program product in a variety of different forms. The program code can be distributed using a computer-readable storage medium having computer-readable program instructions thereon, for enabling a processor to implement aspects of one or more embodiments. The computer-readable storage medium, which is inherently non-transitory, may include volatile and non-volatile and removable and non-removable tangible media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program modules, or other data. The computer-readable storage medium may further include RAM, ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other solid-state memory technology, portable compact disk read-only memory (CD-ROM) or other optical storage device, cassette, tape, magnetic disk storage device or other magnetic storage device, or any other medium that can be used to store the desired information and can be read by a computer. The computer-readable program instructions may be downloaded from the computer-readable storage medium to a computer, another type of programmable data processing device, or another device via a network, or to an external computer or external storage device.

[0096] Computer-readable program instructions stored in a computer-readable medium can be used to guide a computer, other types of programmable data processing devices, or other devices to operate in a particular manner, so that the instructions stored in the computer-readable medium produce an article of manufacture, including instructions that implement the functions, actions, and / or operations specified in the flowchart or diagram. In certain alternative embodiments, the functions, actions, and / or operations specified in the flowchart and diagram can be reordered, processed serially, and / or processed in parallel in accordance with one or more embodiments. In addition, any of the flowchart and / or diagram may include more or fewer nodes or boxes than those illustrated in accordance with one or more embodiments.

[0097] A process, method or algorithm may be embodied in whole or in part using suitable hardware components (such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a state machine, a controller or other hardware components or devices) or a combination of hardware, software and firmware components.

[0098] Figure 6A schematic diagram of a control system 502 configured to control a vehicle 600, which may be an at least partially autonomous vehicle or an at least partially autonomous robot, is depicted. The vehicle 600 includes an actuator 504 and a sensor 506. The sensor 506 may include one or more video sensors, cameras, radar sensors, ultrasonic sensors, lidar sensors, and / or location sensors (e.g., GPS). One or more of the one or more specific sensors may be integrated into the vehicle 600. In the context of sign recognition and processing as described herein, the sensor 506 is a camera mounted to or integrated into the vehicle 600. Alternatively or in addition to one or more specific sensors identified above, the sensor 506 may include a software module that is configured to determine the state of the actuator 504 when executed. A non-limiting example of a software module includes a weather information software module that is configured to determine the classification of weather close to the vehicle 600 or other location.

[0099] The classifier 514 of the control system 502 of the vehicle 600 may be configured to detect objects near the vehicle 600 depending on the input signal x. In such an embodiment, the output signal y may include information characterizing the objects near the vehicle 600. The actuator control command 510 may be determined based on the information. The actuator control command 510 may be used to avoid collision with the detected object.

[0100] In embodiments where the vehicle 600 is at least partially autonomous, the actuator 504 may be embodied in the brakes, propulsion system, engine, drive train, or steering of the vehicle 600. Actuator control commands 510 may be determined such that the actuator 504 is controlled such that the vehicle 600 avoids colliding with a detected object. The detected objects may also be classified according to what the classifier 514 thinks they are most likely to be, such as a pedestrian or a tree. Depending on the classification, the actuator control commands 510 may be determined. In scenarios where a hostile attack may occur, the system described above may be further trained to better detect objects or identify changes in lighting conditions or angles of sensors or cameras on the vehicle 600.

[0101] In other embodiments where the vehicle 600 is an at least partially autonomous robot, the vehicle 600 may be a mobile robot configured to perform one or more functions, such as flying, swimming, diving, and walking. The mobile robot may be an at least partially autonomous lawn mower or an at least partially autonomous cleaning robot. In such embodiments, the actuator control commands 510 may be determined such that a propulsion unit, a steering unit, and / or a braking unit of the mobile robot may be controlled such that the mobile robot may avoid colliding with the identified object.

[0102] In another embodiment, the vehicle 600 is an at least partially autonomous robot in the form of a gardening robot. In such an embodiment, the vehicle 600 may use an optical sensor as a sensor 506 to determine the state of plants in the environment close to the vehicle 600. The actuator 504 may be a nozzle configured to spray a chemical. Depending on the identified species and / or the identified state of the plant, the actuator control command 510 may be determined to cause the actuator 504 to spray the plant with an appropriate amount of an appropriate chemical.

[0103] The vehicle 600 may be an at least partially autonomous robot in the form of a household appliance. Non-limiting examples of household appliances include a washing machine, a stove, an oven, a microwave, or a dishwasher. In such a vehicle 600, the sensor 506 may be an optical sensor configured to detect the state of an object to be processed by the household appliance. For example, in the case where the household appliance is a washing machine, the sensor 506 may detect the state of the laundry inside the washing machine. The actuator control command 510 may be determined based on the detected state of the laundry.

[0104] Figure 7 A schematic diagram of a control system 502 of a system 700 (e.g., a manufacturing machine) configured to control a manufacturing system 702 (e.g., a portion of a production line), such as a punch cutter, a cutter, or a gun drill, is depicted. The control system 502 may be configured to control an actuator 504 configured to control the system 700 (e.g., a manufacturing machine).

[0105] The sensor 506 of the system 700 (e.g., a manufacturing machine) may be an optical sensor configured to capture one or more properties of the manufactured product 704. The classifier 514 may be configured to determine a state of the manufactured product 704 from the one or more captured properties. The actuator 504 may be configured to control the system 700 (e.g., a manufacturing machine) for subsequent manufacturing steps of the manufactured product 704 depending on the determined state of the manufactured product 704. The actuator 504 may be configured to control the function of the system 700 (e.g., a manufacturing machine) on the subsequent manufactured product 706 of the system 700 (e.g., a manufacturing machine) depending on the determined state of the manufactured product 704.

[0106] Figure 8 A schematic diagram of a control system 502 configured to control a power tool 800 (such as a drill or driver) having an at least partially autonomous mode is depicted. The control system 502 may be configured to control an actuator 504 that is configured to control the power tool 800.

[0107] The sensor 506 of the power tool 800 can be an optical sensor configured to capture one or more properties of the working surface 802 and / or the fastener 804 being driven into the working surface 802. The classifier 514 can be configured to determine the state of the working surface 802 and / or the state of the fastener 804 relative to the working surface 802 from the one or more captured properties. The state can be that the fastener 804 is flush with the working surface 802. Alternatively, the state can be the hardness of the working surface 802. The actuator 504 can be configured to control the power tool 800 so that the driving function of the power tool 800 is adjusted depending on the determined state of the fastener 804 relative to the working surface 802 or one or more captured properties of the working surface 802. For example, if the state of the fastener 804 is flush with the working surface 802, the actuator 504 may no longer continue the driving function. As another non-limiting example, the actuator 504 can apply additional or less torque depending on the hardness of the working surface 802.

[0108] Fig. 9 A schematic diagram of a control system 502 configured to control an automated personal assistant 900 is depicted. The control system 502 may be configured to control an actuator 504 configured to control the automated personal assistant 900. The automated personal assistant 900 may be configured to control a household appliance, such as a washing machine, stove, oven, microwave, or dishwasher.

[0109] Sensor 506 may be an optical sensor and / or an audio sensor. The optical sensor may be configured to receive a video image of user 902's gesture 904. The audio sensor may be configured to receive a voice command of user 902.

[0110] The control system 502 of the automated personal assistant 900 can be configured to determine an actuator control command 510 configured for the control system 502. The control system 502 can be configured to determine the actuator control command 510 based on the sensor signal 508 of the sensor 506. The automated personal assistant 900 is configured to transmit the sensor signal 508 to the control system 502. The classifier 514 of the control system 502 can be configured to execute a gesture recognition algorithm to identify the gesture 904 made by the user 902, determine the actuator control command 510, and transmit the actuator control command 510 to the actuator 504. The classifier 514 can be configured to retrieve information from the non-volatile storage device in response to the gesture 904 and output the retrieved information in a form suitable for receipt by the user 902.

[0111] Fig.10A schematic diagram of a control system 502 configured to control a monitoring system 1000 is depicted. The monitoring system 1000 can be configured to physically control access through a door 1002. A sensor 506 can be configured to detect scenes relevant to deciding whether access is authorized. The sensor 506 can be an optical sensor configured to generate and transmit image and / or video data. The control system 502 can use such data to detect the face of a person.

[0112] The classifier 514 of the control system 502 of the monitoring system 1000 can be configured to interpret the image and / or video data by matching the identities of known persons stored in the non-volatile storage device 516 to determine the identity of the person. The classifier 514 can be configured to generate an actuator control command 510 in response to the interpretation of the image and / or video data. The control system 502 is configured to transmit the actuator control command 510 to the actuator 504. In this embodiment, the actuator 504 can be configured to lock or unlock the door 1002 in response to the actuator control command 510. In other embodiments, non-physical logical access control is also possible.

[0113] The monitoring system 1000 may also be a surveillance system. In such an embodiment, the sensor 506 may be an optical sensor configured to detect a scene under surveillance, and the control system 502 is configured to control the display 1004. The classifier 514 is configured to determine a classification of the scene, for example, whether the scene detected by the sensor 506 is suspicious. The control system 502 is configured to: transmit an actuator control command 510 to the display 1004 in response to the classification. The display 1004 may be configured to adjust the displayed content in response to the actuator control command 510. For example, the display 1004 may highlight objects that the classifier 514 considers suspicious. Using embodiments of the disclosed system, the surveillance system can identify such objects and assign categories or descriptors to them.

[0114] Fig.11 A schematic diagram of a control system 502 is depicted, which is configured to control an imaging system 1100, such as an MRI device, an x-ray imaging device, or an ultrasound device. For example, the sensor 506 can be an imaging sensor. The classifier 514 can be configured to determine a classification of all or part of the sensed image. The classifier 514 can be configured to determine or select an actuator control command 510 in response to the classification obtained by the trained neural network. For example, the classifier 514 can interpret an area of ​​the sensed image as a potential abnormality and assign a category or descriptor to the sensed image. In this case, the actuator control command 510 can be determined or selected to cause the display 1102 to display the imaging and highlight the potential abnormal area.

[0115] Although exemplary embodiments are described above, it is not intended that these embodiments describe all possible forms covered by the claims. The words used in this specification are descriptive words, not restrictive words, and it should be understood that various changes can be made without departing from the spirit and scope of the present disclosure. As previously described, the features of various embodiments can be combined to form other embodiments of the present invention, and these other embodiments may not be explicitly described or illustrated. Although various embodiments may have been described as providing advantages or being superior to other embodiments or prior art implementations with respect to one or more desired characteristics, it is recognized by those of ordinary skill in the art that one or more features or characteristics can be compromised to achieve the desired overall system properties, depending on the specific application and implementation. These properties may include, but are not limited to, cost, strength, durability, life cycle cost, marketability, appearance, packaging, size, applicability, weight, manufacturability, ease of assembly, etc. As such, to the extent that any embodiment is described as less desirable than other embodiments or prior art implementations with respect to one or more characteristics, these embodiments are not outside the scope of the present disclosure and may be desirable for specific applications.

Claims

1. A computer-implemented method for pre-training a machine learning network, the computer-implemented method comprising the following steps: (i) receiving a plurality of input images comprising a plurality of pixels; (ii) generating a vision matrix using a plurality of input images and an image encoder of a machine learning network, wherein the vision matrix comprises a list of encoded images; (iii) receiving a plurality of text prompts; (iv) selecting a first text prompt from the plurality of text prompts; (v) sending a first one of the text prompts to a large language model (LLM) to generate a list of candidate word-grams, wherein the list of candidate word-grams is generated by selecting a subset from each word-gram associated with the first one of the text prompts, wherein the subset includes a most probable word-gram associated with the first one of the text prompts, wherein the most probable word-gram is computed responsive to an output of the LLM; (vi) selecting one or more word-grams from the candidate list; (vii) converting one of the text prompts into an updated text prompt by appending one or more selected word-grams associated with the plurality of text prompts; (viii) generating a text matrix using both (1) an updated text prompt including one or more word-grams and (2) a text encoder of a machine learning network, wherein the text matrix includes a list of encoded visual descriptors including the updated text prompt having the one or more word-grams; (ix) multiplying the textual matrix and the visual matrix to generate an image-text similarity matrix, wherein the image-text similarity matrix assigns a numerical value indicating a similarity between each encoded visual descriptor and each encoded image, wherein the similarity is indicated by an entry of the image-text similarity matrix having a numerical value; (x) determining a score associated with the image-text similarity matrix using the numerical values ​​assigned at the image-text similarity matrix; and (xi) when the score drops below a threshold, repeating steps (vi-xi) for a second word-gram of a first one of the text prompts, and when the score exceeds the threshold, adding the one or more word-grams to an updated text prompt, and repeating steps (iv-xi) for the remainder of each of the plurality of text prompts; as well as (xii) after evaluating each of the plurality of text prompts, in response to identifying a highest score associated with a final word-gram, outputting a final word-gram to an updated text prompt. 2 . The method of claim 1 , wherein determining a score associated with the image-text similarity matrix comprises utilizing KL-divergence.

3. The method of claim 1, wherein the image-text similarity matrix is ​​a one-hot encoded matrix of each image.

4. The method of claim 1, wherein the text encoder is a contrastive language-image pretraining (CLIP) text encoder and the image encoder is a CLIP image encoder.

5. The method of claim 1, wherein parameters associated with the image encoder and the text encoder are not modified. The method according to claim 1 , wherein the threshold is the number of iterations.

7. The method of claim 1, wherein the threshold comprises a length associated with the plurality of text prompts. The method of claim 1 , wherein the threshold is a convergence threshold.

9. The method of claim 1, wherein one of the textual prompts is associated with a category representing one of the plurality of input images.

10. The method of claim 1, wherein the one or more word-grams comprise terms.

11. A system comprising: A processor programmed to: (i) receiving a plurality of input images indicating radar, sonar, video, picture, sound or lidar information; (ii) generating a vision matrix using the plurality of input images and an image encoder of the machine learning network, wherein the vision matrix comprises a list of encoded images; (iii) receiving a plurality of text prompts; (iv) selecting a first text prompt from the plurality of text prompts; (v) sending a first one of the text prompts to a large language model (LLM) to generate a list of candidate word-grams, wherein the list of candidate word-grams is generated by selecting a subset of word-grams from each word-gram associated with the first one of the text prompts, wherein the subset includes a most probable word-gram associated with the first one of the text prompts, wherein the most probable word-gram is computed in response to an output of the LLM; (vi) selecting one or more word-grams from the candidate list; (vii) converting one of the text prompts into an updated text prompt by appending one or more selected word-grams associated with the plurality of text prompts; (viii) generating a text matrix using both (1) an updated text prompt including one or more word-grams and (2) a text encoder of a machine learning network, wherein the text matrix includes a list of encoded visual descriptors including the updated text prompt having the one or more word-grams; (ix) multiplying the textual matrix and the visual matrix to generate an image-text similarity matrix, wherein the image-text similarity matrix assigns a numerical value indicating a similarity between each encoded visual descriptor and each encoded image, wherein the similarity is indicated by an entry of the image-text similarity matrix having a numerical value; (x) determining a score associated with the image-text similarity matrix using the numerical values ​​assigned at the image-text similarity matrix; (xi) when the score drops below a threshold, repeating steps (vi-xi) for a second word-gram of a first one of the text prompts, and when the score exceeds the threshold, adding the one or more word-grams to an updated text prompt, and repeating steps (iv-xi) for the remainder of each of the plurality of text prompts; and (xii) after evaluating each of the plurality of text prompts, in response to identifying a highest score associated with a final word-gram, outputting a final word-gram to an updated text prompt.

12. The system of claim 11, wherein the one or more word-grams comprise terms.

13. The system of claim 11, wherein adding the one or more word-grams to the updated text prompt comprises replacing a previous word-gram with a lower score.

14. The system of claim 11, wherein parameters associated with the image encoder and the text encoder remain fixed.

15. A computer-implemented method comprising: (i) receiving a plurality of input images; (ii) generating a vision matrix using the plurality of input images and an image encoder of the machine learning network, wherein the vision matrix comprises a list of encoded images; (iii) receiving a plurality of text prompts; (iv) selecting a first text prompt from the plurality of text prompts; (v) sending a first one of the textual cues to a language model (LM) to generate a list of candidate word-grams, wherein the list of candidate word-grams is a subset of less than all word-grams associated with the first one of the textual cues; (vi) selecting one or more word-grams from the candidate list; (vii) converting one of the text prompts into an updated text prompt by appending one or more selected word-grams associated with the plurality of text prompts; (viii) generating a text matrix using both (1) an updated text prompt including one or more word-grams and (2) a text encoder of a machine learning network, wherein the text matrix includes a list of encoded visual descriptors including the updated text prompt having the one or more word-grams; (ix) multiplying the textual matrix and the visual matrix to generate an image-text similarity matrix, wherein the image-text similarity matrix assigns a numerical value indicating a similarity between each encoded visual descriptor and each encoded image, wherein the similarity is indicated by an entry of the image-text similarity matrix having a numerical value; (x) determining a score associated with the image-text similarity matrix using the numerical values ​​assigned at the image-text similarity matrix; and (xi) when the score drops below a threshold, repeating steps (vi-xi) for a second word-gram of the first one of the text prompts, and when the score exceeds the threshold, adding the one or more word-grams to the updated text prompt, and repeating steps (iv-xi) for the remainder of each of the plurality of text prompts.

16. The method of claim 15, wherein adding the one or more word-grams to the updated text prompt comprises replacing previous word-grams.

17. A method according to claim 15, wherein the list of candidate word-grams is generated by selecting a subset of word-grams from each word-gram associated with a first one of the text prompts, wherein the subset includes the most probable word-gram associated with the first one of the text prompts, and wherein the most probable word-gram is calculated in response to the output of the LLM.

18. The method of claim 15, wherein the method comprises, after evaluating each of the plurality of text prompts, in response to identifying a highest score associated with a final word-gram, outputting a final word-gram to an updated text prompt. The method of claim 15 , wherein the one or more word-grams comprise terms.

20. The method of claim 15, wherein adding the one or more word-grams to the updated text prompt comprises replacing a previous word-gram with a lower score.

Images