Systems and methods for utilizing multiple descriptive features for robust few-sample image learning

Abstract

A system including a machine learning network includes a controller configured to determine a machine learning network using a numerical value assigned at an image-text similarity matrix output at the machine learning network including a text encoder and an image encoder, updating parameters of an untrained layer of the machine learning network with sparse logistic regression to generate a sparse logistic regression layer wherein the image-text similarity matrix is associated with a plurality of input images received at the controller, freezing one or more entries of the sparse logistic regression layer comprising zero values, executing a plurality of input images at one or more unfrozen entries at (1) the image encoder and (2) the sparse logistic regression layer, and in response to execution of the plurality of input images, updating parameters of the image encoder and parameters associated with the one or more unfrozen entries, and outputting the adjusted machine learning model until a threshold value is satisfied.

Description

Technical Field

[0001] The present disclosure relates to a machine learning network, including a machine learning network that uses a few-sample image for learning. Background Art

[0002] Self-supervised visual language models (VLMs) such as contrastive language-image pre-training (CLIP) can create aligned image and text encoders via contrastive training. Unlike traditionally trained classification networks, this alignment achieves zero-shot image classification by prompting the text encoder with hand-crafted inputs such as “photo of {}” and then predicting the target via the maximum inner product using the input image embedding. However, selecting effective prompts for zero-shot learning remains largely an ad-hoc process: some systems have added several prompts such as “cartoon {}” or “art of {}” with the aim of improving ImageNet-R performance and thus also standard ImageNet accuracy. This has led to work that attempts to automatically extract relevant prompts from language models, including work that uses these models to extract multiple visual descriptors and then uses the average prediction of these visual descriptions to classify images.

[0003] However, in the few-shot setting, where a small amount of training data is available, a number of techniques can further improve classifier performance beyond the zero-shot cue alone. For example, it has become commonplace to fine-tune the zero-shot classifier via linear probing or other methods, including methods that interpolate between zero-shots and fine-tuned classifiers to achieve better out-of-distribution robustness. Alternatively, the few-shot data can also be used to adapt the cue itself, for example using techniques from soft cue conditioning, although these learned cue are not human-readable nor are they their nearest dictionary projections. Finally, recent work has also investigated ways to combine automatically extracted cues using few-shot learning, although this approach uses very specific learned weights on this description and its performance lags behind simple linear probing even for a small number of examples. Summary of the invention

[0004] The first illustrative embodiment describes a computer-implemented method for adjusting a pre-trained machine learning network. The method includes: receiving a plurality of input images; receiving a plurality of text data including image labels, the plurality of text data being sent to a large language model (LLM) of the pre-trained machine learning network; generating a plurality of visual descriptors using the LLM and the plurality of text data; generating a text matrix using the plurality of visual descriptors and a text encoder of the machine learning network, wherein the text matrix includes a list of encoded visual descriptors; generating a visual matrix using the plurality of input images and an image encoder of the machine learning network, wherein the visual matrix includes a list of encoded images; multiplying the text matrix and the visual matrix to generate an image-text similarity matrix, wherein the image-text similarity matrix assigns a numerical value indicating the similarity of each encoded visual descriptor to each encoded image. The invention relates to a method for adjusting a machine learning model according to the present invention. The method comprises the steps of: determining similarity between images of a code, wherein the similarity is indicated by entries having non-zero values ​​of an image-text similarity matrix; adjusting a machine learning network by updating parameters of an untrained layer of the machine learning network using sparse logistic regression to generate a sparse logistic regression layer using numerical values ​​assigned at the image-text similarity matrix; freezing one or more entries of the sparse logistic regression layer including zero values; running a plurality of input images at (1) an image encoder and (2) one or more unfrozen entries at the sparse logistic regression layer; and updating parameters of the image encoder and parameters associated with the one or more unfrozen entries in response to the running of the plurality of input images; and outputting the adjusted machine learning model after repeating certain steps until a threshold is met.

[0005] A second illustrative embodiment describes a computer-implemented method for conditioning a pre-trained machine learning network. The computer-implemented method includes the following steps: generating a text matrix using one or more visual descriptors associated with a large language model (LLM) of the network and a text encoder of the machine learning network, wherein the text matrix includes a list of encoded visual descriptors; generating a visual matrix using one or more input images and an image encoder of the machine learning network, wherein the visual matrix includes a list of encoded images; multiplying the text matrix and the visual matrix to generate an image-text similarity matrix, wherein the image-text similarity matrix assigns a numerical value indicating a similarity between each encoded visual descriptor and each encoded image, wherein the similarity is determined by the image-text One or more entries of a similarity matrix having non-zero values ​​are indicated; using the numerical values ​​assigned at the image-text similarity matrix, using sparse logistic regression to update the parameters of an untrained layer of a machine learning network to generate a sparse logistic regression layer; freezing one or more entries of the sparse logistic regression layer including zero values; running multiple input images at (1) an image encoder and (2) one or more unfrozen entries at the sparse logistic regression layer; and in response to the running of the multiple input images, updating the parameters of the image encoder and the parameters associated with the one or more unfrozen entries; and outputting the adjusted machine learning model after running the images again and updating the parameters until a threshold is met.

[0006] A third illustrative embodiment describes a system including a machine learning network, comprising: a controller configured to: update parameters of an untrained layer of the machine learning network using sparse logistic regression to generate a sparse logistic regression layer using numerical values ​​assigned at an image-text similarity matrix output at a machine learning network including a text encoder and an image encoder, wherein the image-text similarity matrix is ​​associated with a plurality of input images received at the controller; freeze one or more entries of the sparse logistic regression layer including zero values; run the plurality of input images at (1) the image encoder and (2) one or more unfrozen entries at the sparse logistic regression layer; and in response to the running of the plurality of input images, update parameters of the image encoder and parameters associated with the one or more unfrozen entries; and output an adjusted machine learning model until a threshold is met. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 A system for training a neural network according to an embodiment is shown.

[0008] Figure 2 A computer-implemented method for training and utilizing a neural network is shown according to one embodiment.

[0009] Figure 3A block diagram according to an embodiment of a machine learning network utilizing a sparse regression layer is illustrated.

[0010] Figure 4 A flow diagram illustrating an embodiment utilizing a sparse regression layer.

[0011] Figure 5 A schematic diagram depicting the interaction between a computer-controlled machine and a control system according to an embodiment is depicted.

[0012] Figure 6 A device configured to control a vehicle according to an embodiment is depicted. Figure 5 Schematic diagram of a control system of a vehicle, which can be a partially autonomous vehicle, a fully autonomous vehicle, a partially autonomous robot, or a fully autonomous robot.

[0013] Figure 7 Depicted Figure 5 Schematic diagram of a control system configured to control a manufacturing machine, such as a punch cutter, a cutting machine, or a gun drill, of a manufacturing system (such as part of a production line).

[0014] Figure 8 Depicted Figure 5 Schematic diagram of a control system configured to control a power tool, such as a drill or driver, having an at least partially autonomous mode.

[0015] Fig. 9 Depicted Figure 5 Schematic diagram of a control system configured to control an automated personal assistant.

[0016] Fig.10 Depicted Figure 5 Schematic diagram of a control system configured to control a monitoring system, such as a controlled access system or a surveillance system.

[0017] Fig.11 Depicted Figure 5 Schematic diagram of a control system configured to control an imaging system, such as an MRI device, an x-ray imaging device, or an ultrasound device. DETAILED DESCRIPTION

[0018] Embodiments of the present disclosure are described herein. However, it is to be understood that the disclosed embodiments are merely examples, and other embodiments may take various and alternative forms. The drawings are not necessarily drawn to scale; some features may be exaggerated or minimized to show the details of a particular component. Therefore, the specific structural and functional details disclosed herein should not be interpreted as limiting, but merely as a representative basis for teaching those skilled in the art to adopt the embodiments in various ways. As will be understood by those of ordinary skill in the art, the various features illustrated and described with reference to any one of the drawings may be combined with the features illustrated in one or more other drawings to produce embodiments that are not explicitly illustrated or described. The combination of the illustrated features provides a representative embodiment of a typical application. However, for a particular application or implementation, various combinations and modifications of features consistent with the teachings of the present disclosure may be desired.

[0019] As used herein, "a," "an," and "the" refer to both singular and plural referents unless the context clearly dictates otherwise. By way of example, a "processor" programmed to perform various functions refers to one processor programmed to perform each function, or more than one processor collectively programmed to perform each of the various functions.

[0020] GPT-2 and 3 may include an in-context learning paradigm that applies LLM to downstream tasks, in which a prompt is provided to elicit a response or action, and then optionally several demonstrations are provided in that context.

[0021] Discrete prompt conditioning automatic prompts search for fixed-length prompts in a token-by-token greedy manner. For each token position, they first select the top k candidate tokens that are most aligned with the gradient of the log-likelihood while fixing the prompt; then, they select the candidate that maximizes the log-likelihood from the k candidates. The first step requires one forward backward pass of the model, and the second step takes k forward steps. This greedy algorithm generates readable but unintelligible prompts. For example, they generate the following prompt for a sentiment analysis task: "a real joy.atmosphere a lot dialogueClone totally[MASK]". Guo et al. used Q-learning to generate prompts, but still lacked comprehensibility. Gao et al. used T5 to generate prompt templates containing training pairs and selected the best template based on the training data. Recently, Shi et al. proposed sampling human-readable prompts based on Langevin dynamics incorporated with fluency constraints. There is also work that suggests hint engineering via edit-based enumeration, data mining, and even LLM itself.

[0022] Continuous Prompt Tuning optimizes on a continuous text embedding space, typically by gradient descent on the training data while keeping the model parameters fixed. These methods are generally faster and more scalable than discrete prompt tuning, although the searched prompts are not semantically meaningful.

[0023] VLM vs. Cue conditioning in VLM aims to minimize the contrast loss between matching image-text pairs. Let the image embedding be Text embedding as Without loss of generality (WLOG), let the first entry of the embedding be the [CLS] token, denoted by g(t)0. The predicted probability is then expressed as: where t c is a zero-shot textual hint for category c. The category whose hint has the largest inner product with the image embedding will be the zero-shot prediction. T The best hint is optimized in the embedding space of . Several subsequent works have proposed various hint adjustment methods for different task settings. Finally, g(t c)0 is essentially a regularized linear exploration in the sense that the search space is constrained by the co-domain of g(·)0. Chen et al. use image embeddings f1,…,f M +1 local information to optimize the optimal transfer distance between local image information and cues. Lu et al. learn a distribution over cues for efficient adaptation to downstream recognition tasks. Wen et al. discuss discrete cue search in the context of a text-to-image setting.

[0024] Pratt et al. prompt the LLM with descriptions for each category and show that these prompts can achieve better zero-shot image classification accuracy. Menon and Vondrick prompt the LLM to generate visual descriptors for image classification. For each category c, they query GPT-3 with the prompt "What are some useful features for distinguishing {c} in photos?". Given an image x, estimate the score for c: where D(c) is the number of descriptors for c, and φ(d;x)=<Encode(d);Encode(x)> is the grounding between image and text features. They show that this average ensemble can outperform the zero-shot classifier while maintaining interpretability.

[0025] Some systems may consider per-category level descriptions in a few-shot setting. A key difference is that such systems perform per-category level description filtering through submodule optimization, and they use a softmax-ed linear weight σ(W) to integrate selected features. On the other hand, the systems and methods described below use sparse logistic regression to directly select features. This approach immediately gives both important features and coefficients, and is statistically optimal under certain sparsity assumptions. One of the potential disadvantages of existing systems is that their visual descriptions are filtered per-category level, which may hinder feature sharing between categories. This partly explains why on ImageNet, when k>=4, such systems may begin to perform worse than linear detection. Since such systems focus on interpretability, they can force σ(W) for probabilistic interpretation, while the following embodiments can focus on robustness and only require W to be sparse. In one embodiment, the system can employ WISE-FT to improve out-of-distribution accuracy.

[0026] Let the fine-tuned weights be Φlearned and let the zero-shot predictor be Φ zs . The system can be observed: although Φlearned is better than Φ on the in-distribution task zsperforms better, but it performs worse on out-of-distribution tasks. Therefore, the system can propose to interpolate these two sets of weights to αΦlearned+(1-α)Φ zs This surprisingly simple weight integration helps both in-distribution and out-of-distribution tasks. This approach also naturally applies to linear detection by simply freezing the CLIP encoder throughout and only training and interpolating the linear head.

[0027] Contrastive training allows CLIP to perform zero-shot classification using prompts such as “photo of {label}”. It can be beneficial to reason about the category of an image with more detailed semantic features (such as “does it have fur” or “does it have a nose”?). This reasoning tells us not only which category the image belongs to, but also why it belongs to that category. Arguably, classifying images using multiple descriptive features may be more intuitive and more interpretable to humans. The contrastive learning process does not explicitly tell the model why the image belongs to a certain category, and the semantic differences between categories are still retained in the image embedding space. Some of the semantic differences may even be abstract ideas that are unlikely to be visually discernible from the image. This phenomenon can be illustrated with the following example below. The system can randomly sample 500 images of airplanes and 500 images of cars from the CIFAR-10 dataset and compute their cosine similarity using the following 5 prompts: 1. “It has wings” 2. “It has four doors” 3. "Photo of an airplane" 4. "Photos of cars" 5. “A crash is fatal” The cosine similarity can be shown and plotted. Even though contrastive training does not necessarily associate cues 1, 2, and 5 with the image itself, there may be a clear separation between the classes using these cues. A particularly interesting cue is “crashes are fatal” because its intuition matches the human intuition that plane crashes are more dangerous than car crashes. Even though cues 1, 2, and 5 are weaker separators compared to cues 3 and 4, this observation leaves open the possibility that we can integrate many weak features to obtain a good classifier.

[0028] Modern image classification is based on directly predicting model classes via large discriminative networks, which makes it difficult to evaluate intuitive visual "features" that may underlie classification decisions. Meanwhile, recent work in joint visual language models (such as CLIP) provides ways to specify natural language descriptions of image categories, but generally focuses on providing a single description for each category. In this work, illustrative embodiments demonstrate that an alternative approach (arguably more similar to our understanding of multiple "visual features" per category) can also provide compelling performance in a robust few-shot learning setting. In particular, the system can automatically enumerate multiple visual descriptions of each category via a large language model (LLM), and then use a visual-image model to convert these descriptions into a set of multiple visual features for each image. The system and method can ultimately use sparse logistic regression to select a relevant subset of these features to classify each image. This provides an "intuitive" set of relevant features for each category, and in a few-shot learning setting, outperforms standard methods such as linear probing. When combined with fine-tuning, this approach is able to outperform existing state-of-the-art fine-tuning methods in both in-distribution performance and out-of-distribution performance.

[0029] Reference is now made to the embodiments illustrated in the accompanying figures, which may apply these teachings to a machine learning model or a neural network. Figure 1 A system 100 for training a neural network (e.g., a deep neural network) is shown. The system 100 may include an input interface for accessing training data 102 for the neural network. For example, Figure 1 As shown, the input interface may be constituted by a data storage interface 104 that may access training data 102 from a data storage device 106. For example, the data storage interface 140 may be a memory interface or a persistent storage device interface, such as a hard disk or SSD interface, but may also be a personal area network, a local area network, or a wide area network interface, such as a Bluetooth, Zigbee, or Wi-Fi interface, or an Ethernet or fiber optic interface. The data storage device 106 may be an internal data storage device of the system 100, such as a hard disk drive or SSD, but may also be an external data storage device, such as a network accessible data storage device.

[0030] In some embodiments, the data storage device 106 may further include a data representation 108 of an untrained version of the neural network, which the system 100 can access from the data storage device 106. However, it will be appreciated that the training data 102 and the data representation 108 of the untrained neural network can also each be accessed from different data storage devices, such as via different subsystems of the data storage interface 104. Each subsystem can be of the type described above for the data storage interface 104. In other embodiments, the data representation 108 of the untrained neural network can be generated internally by the system 100 based on the design parameters of the neural network, and thus may not be explicitly stored on the data storage device 106. The system 100 can further include a processor subsystem 110, which can be configured to provide an iterated function as a replacement for a layer stack of the neural network to be trained during operation of the system 100. Here, the corresponding layers of the replaced layer stack can have mutually shared weights and can receive the output of the previous layer as input, or, for the first layer of the layer stack, receive the initial activation and a portion of the input of the layer stack. The processor subsystem 110 may be further configured to iteratively train the neural network using the training data 102. Here, the iterations of training performed by the processor subsystem 110 may include a forward propagation portion and a backward propagation portion. The processor subsystem 110 may be configured to perform the forward propagation portion by determining an equilibrium point of the iteration function at which the iteration function converges to a fixed point, in addition to other operations that may be performed that define the forward propagation portion—where determining the equilibrium point includes using a numerical root finding algorithm to find a root solution for the iteration function minus its input, and providing the equilibrium point as a substitute for the output of the stack of layers in the neural network. The system 100 may further include an output interface for outputting a data representation 112 of the trained neural network, which data may also be referred to as trained model data 112. For example, as also Figure 1 As illustrated, the output interface may be comprised of a data storage interface 104, which in these embodiments is an input / output ("IO") interface, via which trained model data 112 may be stored in a data storage device 106. For example, a data representation 108 defining an "untrained" neural network may be at least partially replaced by a digital representation 112 of a trained neural network during or after training, as parameters of the neural network (such as weights, hyperparameters, and other types of neural network parameters) may be adapted to reflect training on the training data 102. This is also Figure 1106 by reference numerals 108, 112, which refer to the same data record on the data storage device 106. In other embodiments, the data representation 112 may be stored separately from the data representation 108 defining the "untrained" neural network. In some embodiments, the output interface may be separate from the data storage interface 104, but may generally be of the type described above for the data storage interface 104.

[0031] The structure of system 100 is an example of a system that can be used to train a pre-trained machine learning network that utilizes the few-shot image learning described herein. Figure 2 Additional structures for operating and training machine learning models are shown in .

[0032] Figure 2 A system for implementing a machine learning model described herein, for example, utilizing a pre-trained machine learning network for few-shot image learning described herein, is depicted. System 200 can be implemented to perform few-shot image learning as described herein. System 200 can include at least one computing system 202. Computing system 202 can include at least one processor 204 operably connected to a memory unit 208. Processor 204 can include one or more integrated circuits that implement the functionality of a central processing unit (CPU) 206. CPU 206 can be a commercially available processing unit that implements an instruction set, such as one of the x86, ARM, Power, or MIPS instruction set families. During operation, CPU 206 can execute stored program instructions retrieved from memory unit 208. The stored program instructions can include software that controls the operation of CPU 206 to perform the operations described herein. In some examples, processor 204 can be a system on a chip (SoC) that integrates the functionality of CPU 206, memory unit 208, network interface, and input / output interface into a single integrated device. Computing system 202 can implement an operating system for managing various aspects of operation. Although in Figure 2 One processor 204, one CPU 206, and one memory 208 are shown in FIG. 1 , but of course, more than one processor 204, more than one CPU 206, and more than one memory 208 may be utilized in the overall system.

[0033] The memory unit 208 may include volatile memory and non-volatile memory for storing instructions and data. Non-volatile memory may include solid-state memory, such as NAND flash memory, magnetic and optical storage media, or any other suitable data storage device that retains data when the computing system 202 is disabled or loses electrical power. Volatile memory may include static and dynamic random access memory (RAM) that stores program instructions and data. For example, the memory unit 208 may store a machine learning model 210 or algorithm, a training data set 212 for the machine learning model 210, and an original source data set 216.

[0034] The computing system 202 may include a network interface device 222 configured to provide communications with external systems and devices. For example, the network interface device 222 may include a wired and / or wireless Ethernet interface defined by the Institute of Electrical and Electronics Engineers (IEEE) 802.11 family of standards. The network interface device 222 may include a cellular communication interface for communicating with a cellular network (e.g., 3G, 4G, 5G). The network interface device 222 may further be configured to provide a communication interface to an external network 224 or a cloud.

[0035] The external network 224 may be referred to as the World Wide Web or the Internet. The external network 224 may establish a standard communication protocol between computing devices. The external network 224 may allow information and data to be easily exchanged between computing devices and the network. One or more servers 230 may communicate with the external network 224.

[0036] The computing system 202 may include an input / output (I / O) interface 220, which may be configured to provide digital and / or analog input and output. The I / O interface 220 is used to transfer information between internal storage devices and external input and / or output devices (e.g., HMI devices). The I / O 220 interface may include an associated circuit system or bus network to transfer information to (multiple) processors and storage devices or between (multiple) processors and storage devices. For example, the I / O interface 220 may include digital I / O logic lines that can be read or set by (multiple) processors, handshake lines for supervising data transfer via I / O lines; timing and counting facilities, and other structures known to provide such functions. Examples of input devices include keyboards, mice, sensors, etc. Examples of output devices include monitors, printers, speakers, etc. The I / O interface 220 may include additional serial interfaces (e.g., universal serial bus (USB) interfaces) for communicating with external devices.

[0037] The computing system 202 may include a human-machine interface (HMI) device 218, which may include any device that enables the system to receive control inputs. Examples of input devices may include human-machine interface inputs, such as keyboards, mice, touch screens, voice input devices, and other similar devices. The computing system 202 may include a display device 232. The computing system 202 may include hardware and software for outputting graphics and text information to the display device 232. The display device 232 may include an electronic display screen, a projector, a printer, or other suitable devices for displaying information to a user or operator. The computing system 202 may be further configured to allow interaction with a remote HMI and a remote display device via a network interface device 222.

[0038] The system can be implemented using one or more computing systems. Although this example depicts a single computing system 202 that implements all of the described features, it is intended that the various features and functions can be separated and implemented by multiple computing units that communicate with each other. The specific system architecture selected can depend on various factors.

[0039] The system may implement a machine learning algorithm 210 configured to analyze a raw source data set 216. The raw source data set 216 may include raw or unprocessed sensor data that may represent an input data set for the machine learning system. The raw source data set 216 may include video, video clips, images, text-based information, audio or human speech, time series data (e.g., pressure sensor signals over time), and raw or partially processed sensor data (e.g., radar images of objects). An image may represent a video, or picture data may include multiple pixels that form a scene. A pixel may be the smallest addressable element in a raster image, or the smallest addressable element in a dot matrix display device. In most digital display devices, a pixel may be the smallest element that can be manipulated by software. Each pixel may be a sample of an original image or a synthesized image. In one embodiment, more samples generally provide a more accurate representation of the original. The intensity of each pixel may be variable. Reference Figures 5 to 11 Several different examples of inputs are shown and described. In some examples, the machine learning algorithm 210 can be a neural network algorithm (e.g., a deep neural network) designed to perform a predetermined function. For example, the neural network algorithm can be configured to identify road signs or pedestrians in an image in an automotive application. The machine learning algorithm(s) 210 may include an algorithm configured to operate a pre-trained machine learning network using few-shot image learning as described herein.

[0040] The computer system may store a training data set 212 for the machine learning algorithm 210. The training data set 212 may represent a previously constructed data set for training the machine learning algorithm 210. The machine learning algorithm 210 may use the training data set 212 to learn weighting factors associated with the neural network algorithm. The training data set 212 may include a source data set with corresponding achievements or results that the machine learning algorithm 210 attempts to replicate via the learning process. In this example, the training data set 212 may include an input image that includes an object (e.g., a road sign). The input image may include various scenes in which objects are identified.

[0041] The machine learning algorithm 210 can operate in a learning mode using the training data set 212 as input. The machine learning algorithm 210 can be executed in multiple iterations using data from the training data set 212. With each iteration, the machine learning algorithm 210 can update the internal weighting factors based on the achieved results. For example, the machine learning algorithm 210 can compare the output results (e.g., the reconstructed or supplemented image in the case where the image data is the input) with those output results included in the training data set 212. Since the training data set 212 includes the expected results, the machine learning algorithm 210 can determine when the performance is acceptable. After the machine learning algorithm 210 reaches a predetermined performance level (e.g., 100% consistent with the results associated with the training data set 212) or converges, the machine learning algorithm 210 can be executed using data that is not in the training data set 212. It should be understood that in the present disclosure, "convergence" can mean that a set (e.g., predetermined) number of iterations have occurred, or the residual is small enough (e.g., the change in the approximate probability within the iteration changes by less than a threshold), or other convergence conditions. The trained machine learning algorithm 210 can be applied to a new data set to generate annotation data.

[0042] The machine learning algorithm 210 may be configured to identify specific features in the raw source data 216. The raw source data 216 may include multiple instances or input data sets for which supplementary results are desired. For example, the machine learning algorithm 210 may be configured to identify the presence of road signs in a video image and annotate the presence. The machine learning algorithm 210 may be programmed to process the raw source data 216 to identify the presence of specific features. The machine learning algorithm 210 may be configured to identify the features in the raw source data 216 as predetermined features (e.g., road signs). The raw source data 216 may be derived from a variety of sources. For example, the raw source data 216 may be actual input data collected by the machine learning system. The raw source data 216 may be machine generated for testing the system. As an example, the raw source data 216 may include raw video images from a camera.

[0043] In an example, the original source data 216 may include image data representing an image. Applying the machine learning algorithms described herein (eg, few-shot image learning, CLIP model, etc.), the output may be a conditioned network associated with a collection of images.

[0044] Figure 3 An example of an embodiment of a system overview according to an embodiment is illustrated. Text prompts 301 such as "car", "airplane" and "bird" can be used for one or more tags. In one example, the input to LLM 305 can be a prompt for a visual descriptor 303 for each category (e.g., "give me a list of visual descriptors for cars", "give me a list of visual descriptors for airplanes", etc.). Then, the output of LLM 305 can be a list 307 of visual descriptors for each category (e.g., "cars have bumpers", "airplanes have wings", etc.). These visual descriptors 307 are then fed to a text encoder 309, such as a CLIP text encoder. In one example, a visual descriptor 307 can be "it has four doors", "it has four wheels", "it has feathers", "it has wings", "it can fly" and "it has an overhead bin".

[0045] The image 310 may be fed to an image encoder 312, which is denoted as f. Throughout this description, g(·) may be used to denote the text encoder 307, and f(·) may be used to denote the image encoder 312. The system may use t for text tokens and p for the image 310. For a vector v, the subscript v i represents the i-th entry. The system may make the symbol t c Overload to represent vectors belonging to class c, which should be clear from the context. The system can generate features, select a sparse set of useful descriptions, and finally align the encoder.

[0046] To generate visual descriptors for ImageNet and its variants, the system can first query an LLM 305 (e.g., GPT-3) using the following prompt 303: "Give me a long list of descriptions for {label}:". GPT-3 can be very sensitive to formatting instructions. Using the prompt "Give me a list" can lead to a list format, making it straightforward to select useful text using regular expressions. In such an example, the list can be queried without explicitly asking the LLM (e.g., GPT-3) for visual features. In one example, even abstract text can be used for classification. The system can make these descriptors conditioned on the category name, using a regular expression for each category c and the i-th descriptor has the form "{c}, which has ” For each category c, the system can collect M descriptors from GPT-3.

[0047] In addition, for each category, there is a collection of handcrafted hint templates, such as “photo of {}” or “art by {}”. If the total number of such templates is T, then the system can generate a total of T hint embeddings for each category, given the category name c. In addition to the above visual descriptors, the system can also take the average of these hint embeddings, resulting in M ​​hint embeddings for each category. c +1 number of hints. For simplicity, the system may refer to the text features generated by GPT-3 as visual descriptors (VDs), the templates with category names as category hints, and the union as enhanced visual descriptors (AVDs). The same names may be used to refer to embeddings.

[0048] If the system represents M = ∑ c∈C M c , then the visual descriptor, category hint and enhanced visual descriptor can be encoded into three matrices Given an image embedding These three matrices create three basic sets h vd =U vd z、h cp =U cp z and h avd =U avd z. In one example, U vd can be referred to as the text matrix 311. All three U matrices can be fixed and never trained. Using the basic features h, the system can learn three matrices set up Then W vd U vd Leading to average integration. In one example, h avd It can refer to the image-text similarity matrix. Set W cp =I |C|×|C| , the system can obtain the zero-shot classifier W zs =W cp U cp . It is natural to convert W vd and W cp Merge into W avd =[W vd , W cp ]. avd It can also be called sparse logistic regression layer 317. In one example, all three matrices can be used as a zero-shot classifier. Therefore, Figure 3The matrices shown may be referred to as a text matrix 311 , an image matrix 314 , and an image-text similarity matrix 315 .

[0049] Learning sparse visual descriptor ensembles. The matrix U defined previously avd can be viewed as a linear projection of the image embedding onto an M+|C|-dimensional semantic space. Although this space has a high ambient dimensionality, the projected embedding exists in a low-dimensional manifold whose rank is less than or equal to the rank of the image embedding space. avd To enforce the sparse constraint, the system can avd The most important dimensions are selected. The system can demonstrate that the selected subspace is also robust to natural distribution shifts. Intuitively, a large distribution shift in the image embedding space may correspond to only a small shift in the semantic space, since the semantics of the image should be invariant.

[0050] In fixed U avd In the case of , the system can use l1 regularization ||W avd ||1 Come and learn W avd Sparse logistic regression not only selects important features, but it actually looks for intuitive features as well. On CIFAR-10 (or another dataset of images used to train the network), the system can demonstrate that the selected features are often the ones that actually describe the class. In one embodiment, for each class, the system can pick the 3 features with the largest coefficients.

[0051] In obtaining sparse Afterwards, the system can be fixed The sparsity pattern and U avd , and fine-tune the image encoder f and The system can therefore prompt GPT-3 with a list of visual descriptors for each category and encode these texts. By taking the inner product, the image embedding is based on these descriptors. The image embedding in , which projects it to dimensional space, but it may exist in a submanifold. Sparse logistic regression 317 is applied on the training data for feature selection. Finally, the system can freeze 319 the sparse pattern. Thus, any zero values ​​associated with the sparse logistic regression layer 317 can be frozen. The image can be run through the image encoder 312 with adjusted parameters and using the update matrix with the frozen entries. Such steps can be repeated until a convergence threshold is met, a number of iterations is met, a loss function threshold is met, etc.

[0052] Figure 4A flowchart of an illustrative embodiment using a sparse regression layer is shown. LM can be used as a language model. C can be used as a set of categories. T = {t1, t2, ..., t M} can be used as a set of visual descriptors. f can be used to define an image encoder, such as a CLIP image encoder. g can be used as a text encoder, such as a CLIP text encoder. In one example, d can be used to define the latent dimension of the CLIP model. In another example, X = {x1, x2, ..., x n}: is the training set of images. The set of image labels can be represented as Y = {y1, ..., y n}: , where y i Is for x i The one-hot encoded matrix of each image label can be shown as Right now,

[0053] A training algorithm may be used to train each category c. For each category c, at step 401, the system may set a LM (language model) to generate an M for category c. c (the number of visual descriptors of category c) visual descriptors. M may indicate the total number of visual descriptors (thus M = ∑ c∈C M c ). One way to do this is to prompt LM with the text "Give me a long list of descriptions for {category name}", and select M from the output list. c At step 403, the system may generate a matrix where row i is represented by g(t i ) is generated; this is a list of encoded visual descriptors. At step 405, the system can generate a matrix where row i is f(x i ), which may be a list of coded images. At step 407, the system may T Multiply to get Among them, H ij Represents image x i With the visual descriptor t j The similarity between . Consider column i of this matrix as image x i It may be useful to have a feature set of , where each feature describes how much of the visual descriptor is applicable to the image x i digit. The system can train the parameters of W via sparse logistic regression to fit the equation (i.e., train the entries of the matrix using l1 regularization on the entries of W). The method may include adjusting the sparse logistic regression layer at step 408. Thus, the system may use the numerical values ​​assigned at the image-text similarity matrix to adjust the machine learning network by updating the parameters of the untrained layers of the machine learning network using sparse logistic regression to generate a sparse logistic regression layer. At step 409, the system may fix the sparsity pattern of W, such as forcing the matrix entries equal to 0 to remain 0. This may generate a sparse pattern layer with unfrozen entries. At step 411, the system may fine-tune both the non-zero entries of W and the parameters of the image encoder f. At step 413, the system may run the image set again through an updated image encoder (e.g., an image encoder with updated parameters) and via a sparse logistic regression layer with unfrozen entries (e.g., not using frozen entries). At decision 415, the system may determine whether a threshold is met to determine whether to output a newly adjusted machine learning model. The threshold may be a number of iterations (e.g., an iteration threshold), a convergence threshold, a loss function algorithm, etc. If the threshold is met, the system can output the adjusted machine learning model at step 417. If the threshold is not met, the system can update the image encoder parameters and continue to rerun the image set again using the updated image encoder parameters and the sparse logistic regression layer with unfrozen entries. Thus, the image encoder parameters can be updated until the threshold is met.

[0054] Figure 5 A schematic diagram of the interaction between a computer-controlled machine 500 and a control system 502 is depicted. The computer-controlled machine 500 includes an actuator 504 and a sensor 506. The actuator 504 may include one or more actuators, and the sensor 506 may include one or more sensors. The sensor 506 is configured to sense a condition of the computer-controlled machine 500. The sensor 506 may be configured to encode the sensed condition into a sensor signal 508 and transmit the sensor signal 508 to the control system 502. Non-limiting examples of the sensor 506 include video, radar, lidar, ultrasound, and motion sensors. Image data, such as video images, picture images, radar images, sound images, etc., may be retrieved from these sensors. In one embodiment, the sensor 506 is an optical sensor configured to sense an optical image of the environment near the computer-controlled machine 500.

[0055] The control system 502 is configured to receive sensor signals 508 from the computer controlled machine 500. As explained below, the control system 502 may be further configured to calculate actuator control commands 510 depending on the sensor signals and transmit the actuator control commands 510 to the actuators 504 of the computer controlled machine 500.

[0056] like Figure 5 As shown, the control system 502 includes a receiving unit 512. The receiving unit 512 can be configured to receive the sensor signal 508 from the sensor 506 and transform the sensor signal 508 into an input signal x. In an alternative embodiment, the sensor signal 508 is directly received as the input signal x without the receiving unit 512. Each input signal x can be a part of each sensor signal 508. The receiving unit 512 can be configured to process each sensor signal 508 to generate each input signal x. The input signal x can include data corresponding to the image recorded by the sensor 506.

[0057] The control system 502 includes a classifier 514. The classifier 514 can be configured to classify the input signal x into one or more labels using a machine learning (ML) algorithm (such as the neural network described above). The classifier 514 is configured to be parameterized by parameters such as those described above (e.g., parameter θ). The parameter θ can be stored in and provided by a non-volatile storage device 516. The classifier 514 is configured to determine an output signal y from the input signal x. Each output signal y includes information that assigns one or more labels to each input signal x. The classifier 514 can transmit the output signal y to a conversion unit 518. The conversion unit 518 is configured to convert the output signal y into an actuator control command 510. The control system 502 is configured to transmit the actuator control command 510 to the actuator 504, which is configured to actuate the computer-controlled machine 500 in response to the actuator control command 510. In another embodiment, the actuator 504 is configured to actuate the computer-controlled machine 500 directly based on the output signal y.

[0058] After the actuator 504 receives the actuator control command 510, the actuator 504 is configured to perform an action corresponding to the associated actuator control command 510. The actuator 504 may include control logic configured to transform the actuator control command 510 into a second actuator control command for controlling the actuator 504. In one or more embodiments, the actuator control command 510 may be used to control a display instead of or in addition to the actuator.

[0059] In another embodiment, the control system 502 includes the sensor 506 instead of or in addition to the computer-controlled machine 500 including the sensor 506. The control system 502 may also include the actuator 504 instead of or in addition to the computer-controlled machine 500 including the actuator 504.

[0060] like Figure 5As shown, the control system 502 also includes a processor 520 and a memory 522. The processor 520 may include one or more processors. The memory 522 may include one or more memory devices. The classifier 514 of one or more embodiments (e.g., a machine learning algorithm, such as those described above with respect to an image encoder or a text encoder) may be implemented by the control system 502, which includes a non-volatile storage device 516, a processor 520, and a memory 522.

[0061] The non-volatile storage 516 may include one or more persistent data storage devices, such as a hard drive, an optical drive, a tape drive, a non-volatile solid-state device, a cloud storage device, or any other device capable of persistently storing information. The processor 520 may include one or more devices selected from a high-performance computing (HPC) system, including a high-performance core, a microprocessor, a microcontroller, a digital signal processor, a microcomputer, a central processing unit, a field programmable gate array, a programmable logic device, a state machine, a logic circuit, an analog circuit, a digital circuit, or any other device that manipulates signals (analog or digital) based on computer-executable instructions residing in the memory 522. The memory 522 may include a single memory device or multiple memory devices, including but not limited to random access memory (RAM), volatile memory, non-volatile memory, static random access memory (SRAM), dynamic random access memory (DRAM), flash memory, cache memory, or any other device capable of storing information.

[0062] The processor 520 may be configured to read into the memory 522 and execute computer executable instructions residing in the non-volatile storage 516 and embodying one or more ML algorithms and / or methods of one or more embodiments. The non-volatile storage 516 may include one or more operating systems and applications. The non-volatile storage 516 may store computer programs in compiled and / or interpreted form created using various programming languages ​​and / or techniques, including without limitation and alone or in combination Java, C, C++, C#, Objective C, Fortran, Pascal, JavaScript, Python, Perl, and PL / SQL.

[0063] When executed by the processor 520, the computer executable instructions of the non-volatile storage device 516 may cause the control system 502 to implement one or more of the ML algorithms and / or methods disclosed herein. The non-volatile storage device 516 may also include ML data (including data parameters) that support the functions, features, and processes of one or more embodiments described herein.

[0064] The program code embodying the algorithm and / or method described herein can be distributed individually or collectively as a program product in various different forms. The program code can be distributed using a computer-readable storage medium having a computer-readable program instruction thereon, for causing the processor to perform aspects of one or more embodiments. The computer-readable storage medium, which is essentially non-temporary, may include volatile and non-volatile media and removable and non-removable tangible media implemented by any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data). The computer-readable storage medium may further include RAM, ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other solid-state memory technology, portable compact disk read-only memory (CD-ROM) or other optical storage device, cassette, tape, disk storage device or other magnetic storage device, or any other medium that can be used to store desired information and can be read by a computer. The computer-readable program instruction can be downloaded from the computer-readable storage medium to a computer, another type of programmable data processing device or another device, or downloaded to an external computer or external storage device via a network.

[0065] Computer-readable program instructions stored in a computer-readable medium can be used to instruct a computer, other types of programmable data processing devices, or other devices to operate in a particular manner so that the instructions stored in the computer-readable medium produce an article including instructions that implement the functions, actions, and / or operations specified in the flowchart or figure. In certain alternative embodiments, the functions, actions, and / or operations specified in the flowchart and figure can be reordered, processed serially, and / or processed concurrently in accordance with one or more embodiments. Moreover, any of the flowchart and / or figure may include more or fewer nodes or boxes than those illustrated in accordance with one or more embodiments.

[0066] The process, method or algorithm may be embodied in whole or in part using suitable hardware components, such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a state machine, a controller, or other hardware components or devices, or a combination of hardware, software and firmware components.

[0067] Figure 6A schematic diagram of a control system 502 configured to control a vehicle 600, which may be an at least partially autonomous vehicle or an at least partially autonomous robot, is depicted. The vehicle 600 includes an actuator 504 and a sensor 506. The sensor 506 may include one or more video sensors, cameras, radar sensors, ultrasonic sensors, lidar sensors, and / or location sensors (e.g., GPS). One or more of the one or more specific sensors may be integrated into the vehicle 600. In the context of landmark recognition and processing described herein, the sensor 506 is a camera mounted to or integrated into the vehicle 600. Instead of or in addition to the one or more specific sensors identified above, the sensor 506 may include a software module that is configured to determine the state of the actuator 504 when executed. A non-limiting example of a software module includes a weather information software module that is configured to determine the classification of weather near the vehicle 600 or other location.

[0068] The classifier 514 of the control system 502 of the vehicle 600 may be configured to detect an object in the vicinity of the vehicle 600 depending on the input signal x. In such an embodiment, the output signal y may include information characterizing that the object is in the vicinity of the vehicle 600. The actuator control command 510 may be determined based on this information. The actuator control command 510 may be used to avoid a collision with the detected object.

[0069] In embodiments where the vehicle 600 is at least partially autonomous, the actuator 504 may be embodied in a brake, propulsion system, engine, transmission, or steering system of the vehicle 600. The actuator control command 510 may be determined such that the actuator 504 is controlled such that the vehicle 600 avoids a collision with a detected object. The detected objects may also be classified according to what the classifier 514 thinks they are most likely to be, such as a pedestrian or a tree. The actuator control command 510 may be determined depending on the classification. In situations where adversarial attacks may occur, the above system may be further trained to better detect objects or identify changes in the angle or lighting conditions of the sensors or cameras on the vehicle 600.

[0070] In other embodiments in which the vehicle 600 is an at least partially autonomous robot, the vehicle 600 may be a mobile robot configured to perform one or more functions, such as flying, swimming, diving, and walking. The mobile robot may be an at least partially autonomous lawn mower or an at least partially autonomous cleaning robot. In such embodiments, actuator control commands 510 may be determined such that a propulsion unit, a steering unit, and / or a braking unit of the mobile robot may be controlled such that the mobile robot may avoid a collision with the identified object.

[0071] In another embodiment, the vehicle 600 is an at least partially autonomous robot in the form of a gardening robot. In such an embodiment, the vehicle 600 may use an optical sensor as a sensor 506 to determine the state of plants in the environment near the vehicle 600. The actuator 504 may be a nozzle configured to spray a chemical. Depending on the identified variety and / or the identified state of the plant, the actuator control command 510 may be determined to cause the actuator 504 to spray the plant with an appropriate amount of an appropriate chemical.

[0072] The vehicle 600 may be an at least partially autonomous robot in the form of a household appliance. Non-limiting examples of household appliances include a washing machine, a stove, an oven, a microwave, or a dishwasher. In such a vehicle 600, the sensor 506 may be an optical sensor configured to detect the state of an object to be processed by the household appliance. For example, in the case where the household appliance is a washing machine, the sensor 506 may detect the state of the laundry in the washing machine. The actuator control command 510 may be determined based on the detected state of the laundry.

[0073] Figure 7 A schematic diagram of a control system 502 is depicted, the control system 502 being configured to control a system 700 (e.g., a manufacturing machine), such as a punch cutter, a cutter, or a gun drill, of a manufacturing system 702 (e.g., a portion of a production line). The control system 502 may be configured to control an actuator 504, which is configured to control the system 700 (e.g., a manufacturing machine).

[0074] The sensor 506 of the system 700 (e.g., a manufacturing machine) may be an optical sensor configured to capture one or more properties of the manufactured product 704. The classifier 514 may be configured to determine a state of the manufactured product 704 from one or more of the captured properties. The actuator 504 may be configured to control the system 700 (e.g., a manufacturing machine) depending on the determined state of the manufactured product 704 for subsequent manufacturing steps of the manufactured product 704. The actuator 504 may be configured to control a function of the system 700 (e.g., a manufacturing machine) on a subsequent manufactured product 106 of the system 700 (e.g., a manufacturing machine) depending on the determined state of the manufactured product 704.

[0075] Figure 8 A schematic diagram of a control system 502 configured to control a power tool 800 , such as a drill or driver, having an at least partially autonomous mode is depicted. The control system 502 may be configured to control an actuator 504 configured to control the power tool 800 .

[0076] The sensor 506 of the power tool 800 may be an optical sensor configured to capture one or more properties of the working surface 802 and / or the fastener 804 driven into the working surface 802. The classifier 514 may be configured to determine the state of the working surface 802 and / or the state of the fastener 804 relative to the working surface 802 by one or more of the captured properties. The state may be that the fastener 804 is flush with the working surface 802. Alternatively, the state may be the hardness of the working surface 802. The actuator 504 may be configured to control the power tool 800 so that the driving function of the power tool 800 is adjusted depending on the determined state of the fastener 804 relative to the working surface 802 or one or more captured properties of the working surface 802. For example, if the state of the fastener 804 is flush with the working surface 802, the actuator 504 may interrupt the driving function. As another non-limiting example, the actuator 504 may apply additional or less torque depending on the hardness of the working surface 802.

[0077] Fig. 9 Depicted is a schematic diagram of a control system 502 configured to control an automated personal assistant 900. The control system 502 may be configured to control an actuator 504 configured to control the automated personal assistant 900. The automated personal assistant 900 may be configured to control a household appliance, such as a washing machine, stove, oven, microwave, or dishwasher.

[0078] Sensor 506 may be an optical sensor and / or an audio sensor. The optical sensor may be configured to receive a video image of user 902's gesture 904. The audio sensor may be configured to receive a voice command of user 902.

[0079] The control system 502 of the automated personal assistant 900 can be configured to determine an actuator control command 510 that is configured to control the system 502. The control system 502 can be configured to determine the actuator control command 510 based on the sensor signal 508 of the sensor 506. The automated personal assistant 900 is configured to transmit the sensor signal 508 to the control system 502. The classifier 514 of the control system 502 can be configured to execute a gesture recognition algorithm to identify the gesture 904 made by the user 902, determine the actuator control command 510, and transmit the actuator control command 510 to the actuator 504. The classifier 514 can be configured to retrieve information from the non-volatile storage device in response to the gesture 904, and output the retrieved information in a form suitable for receipt by the user 902.

[0080] Fig.10A schematic diagram of a control system 502 configured to control a monitoring system 1000 is depicted. The monitoring system 1000 can be configured to physically control access through a door 1002. A sensor 506 can be configured to detect scenes relevant to deciding whether to grant access. The sensor 506 can be an optical sensor configured to generate and transmit image and / or video data. Such data can be used by the control system 502 to detect a person's face.

[0081] The classifier 514 of the control system 502 of the monitoring system 1000 can be configured to interpret the image and / or video data by matching the identities of known persons stored in the non-volatile storage device 516 to determine the identity of the person. The classifier 514 can be configured to generate an actuator control command 510 in response to the interpretation of the image and / or video data. The control system 502 is configured to transmit the actuator control command 510 to the actuator 504. In this embodiment, the actuator 504 can be configured to lock the door 1002 or unlock the door 1002 in response to the actuator control command 510. In other embodiments, non-physical logical access control is also possible.

[0082] The monitoring system 1000 may also be a surveillance system. In such an embodiment, the sensor 506 may be an optical sensor configured to detect a scene under surveillance, and the control system 502 is configured to control the display 1004. The classifier 514 is configured to determine the classification of the scene, such as whether the scene detected by the sensor 506 is suspicious. The control system 502 is configured to transmit the actuator control command 510 to the display 1004 in response to the classification. The display 1004 may be configured to adjust the displayed content in response to the actuator control command 510. For example, the display 1004 may highlight objects that the classifier 514 considers suspicious. Using embodiments of the disclosed system, the surveillance system may identify such objects and assign categories or descriptors to them.

[0083] Fig.11 A schematic diagram of a control system 502 configured to control an imaging system 1100 (e.g., an MRI device, an x-ray imaging device, or an ultrasound device) is depicted. The sensor 506 may be, for example, an imaging sensor. The classifier 514 may be configured to determine a classification of all or a portion of a sensed image. The classifier 514 may be configured to determine or select an actuator control command 510 in response to the classification obtained by the trained neural network. For example, the classifier 514 may interpret an area of ​​the sensed image as a potential abnormality and assign a category or descriptor to the sensed image. In this case, the actuator control command 510 may be determined or selected so that the display 1102 displays the imaging and highlights the potential abnormal area.

[0084] Although exemplary embodiments are described above, these embodiments are not intended to describe all possible forms covered by the claims. The words used in the specification are descriptive rather than limiting words, and it is understood that various changes can be made without departing from the spirit and scope of the present disclosure. As previously described, the features of the various embodiments can be combined to form further embodiments of the present invention, which may not be explicitly described or illustrated. Although various embodiments can be described as providing advantages or being preferred over other embodiments or prior art implementations with respect to one or more desired characteristics, it is recognized by those of ordinary skill in the art that one or more features or characteristics can be compromised to achieve the desired overall system properties, depending on the specific application and implementation. These properties may include, but are not limited to, cost, strength, durability, life cycle cost, marketability, appearance, packaging, size, serviceability, weight, manufacturability, ease of assembly, etc. Therefore, insofar as any embodiment is described as not as desirable as other embodiments or prior art implementations with respect to one or more characteristics, these embodiments are not outside the scope of the present disclosure and may be desirable for specific applications.

Claims

1. A computer-implemented method for conditioning a pre-trained machine learning network, the computer-implemented method comprising the following steps: (i) receiving a plurality of input images comprising a plurality of pixels; (ii) receiving a plurality of text data including image tags, wherein the plurality of text data are sent to a large language model (LLM) of the pre-trained machine learning network; (iii) generating a plurality of visual descriptors using the LLM and the plurality of text data; (iv) generating a text matrix using the plurality of visual descriptors and a text encoder of the machine learning network, wherein the text matrix comprises a list of encoded visual descriptors; (v) generating a vision matrix using the plurality of input images and an image encoder of the machine learning network, wherein the vision matrix comprises a list of encoded images; (vi) multiplying the textual matrix and the visual matrix to generate an image-text similarity matrix, wherein the image-text similarity matrix assigns values ​​indicating similarity between each encoded visual descriptor and each encoded image, wherein similarity is indicated by entries of the image-text similarity matrix having non-zero values; (vii) using the values ​​assigned at the image-text similarity matrix, adjusting the machine learning network by updating parameters of an untrained layer of the machine learning network using sparse logistic regression to generate a sparse logistic regression layer; (viii) freezing one or more entries of the sparse logistic regression layer including zero values; (ix) running the plurality of input images through one or more unfrozen entries at (1) the image encoder and (2) the sparse logistic regression layer; as well as (x) updating parameters of the image encoder and parameters associated with one or more unfrozen entries in response to operation of the plurality of input images; as well as (xi) Output the adjusted machine learning model after repeating steps (ix-x) until the threshold is met.

2. The method of claim 1, wherein the text encoder is a contrastive language-image pre-training (CLIP) text encoder and the image encoder is a CLIP image encoder.

3. The method of claim 1, wherein the parameters are updated in response to a neural network optimization algorithm. The method according to claim 1 , wherein the threshold is the number of iterations. The method of claim 1 , wherein the threshold is a convergence threshold.

6. The method of claim 1, wherein the large language model is a Large Language Model Meta AI (LLaMA).

7. The method of claim 1, wherein the image-text similarity matrix is ​​a one-hot encoded matrix of each image.

8. A computer-implemented method for tuning a pre-trained machine learning network, the computer-implemented method comprising the steps of: (i) generating a text matrix using one or more visual descriptors associated with a large language model (LLM) of the network and a text encoder of the machine learning network, wherein the text matrix comprises a list of encoded visual descriptors; (ii) generating a vision matrix using one or more input data indicating images and an image encoder of the machine learning network, wherein the vision matrix comprises a list of encoded images, wherein the images comprise video data, sonar data, audio data, lidar data, or picture data; (iii) multiplying the textual matrix and the visual matrix to generate an image-text similarity matrix, wherein the image-text similarity matrix assigns values ​​indicating similarity between each encoded visual descriptor and each encoded image, wherein similarity is indicated by one or more entries of the image-text similarity matrix having a non-zero value; (iv) using the values ​​assigned at the image-text similarity matrix, using sparse logistic regression to update the parameters of the untrained layer of the machine learning network to generate a sparse logistic regression layer; (v) freezing one or more entries of the sparse logistic regression layer including zero values; (vi) running the plurality of input images through one or more unfrozen entries at (1) the image encoder and (2) the sparse logistic regression layer; as well as (vii) updating parameters of the image encoder and parameters associated with one or more unfrozen entries in response to operation of the plurality of input images; as well as (viii) Outputting the adjusted machine learning model after repeating steps (vi-vii) until the threshold is met.

9. The method of claim 8, wherein the adjusted machine learning model is adjusted relative to the plurality of input images.

10. The method according to claim 8, wherein the input image is video, picture, radar, sonar, sound or other image information.

11. The method of claim 8, wherein the LLM includes more than ten million parameters associated with the LLM, and the parameters associated with the LLM are not updated. The method of claim 8 , wherein the threshold is a convergence threshold.

13. The method of claim 8, wherein the large language model is ChatGPT.

14. The method of claim 8, wherein the image-text similarity matrix is ​​a one-hot encoded matrix of each image.

15. A system comprising a machine learning network, comprising: The processor is programmed to: updating parameters of an untrained layer of the machine learning network using sparse logistic regression using values ​​assigned at an image-text similarity matrix output at the machine learning network including a text encoder and an image encoder to generate a sparse logistic regression layer, wherein the image-text similarity matrix is ​​associated with a plurality of input data indicative of an image received at a controller, wherein the image includes video data, sonar data, audio data, lidar data, or picture data; freezing one or more entries of the sparse logistic regression layer including zero values; Running a plurality of input images through one or more unfrozen entries at (1) the image encoder and (2) the sparse logistic regression layer; as well as In response to the operation of the plurality of input images, updating parameters of the image encoder and parameters associated with one or more unfrozen entries; as well as Output the adjusted machine learning model until the threshold is met.

16. The system of claim 15, wherein the adjusted machine learning model is output after repeatedly updating parameters of the image encoder and parameters associated with one or more unfrozen entries.

17. The system of claim 15, wherein the numerical value indicates a similarity between each encoded visual indicator and each encoded image, wherein similarity is indicated by one or more entries of the image-text similarity matrix having a non-zero value.

18. The system of claim 15, wherein the image-text similarity matrix is ​​a zero-shot classifier.

19. The system of claim 15, wherein both the text encoder and the image encoder are CLIP encoders.

20. The system of claim 15, wherein the processor is further programmed to: in response to the threshold not being met, further adjust a parameter of the image encoder.

Images