Scalable cue learning for large vision-language models

Abstract

A method of generating text-driven cues and category prediction probabilities using a Vision-Language Model (VLM) includes receiving candidate category names associated with a plurality of candidate categories of an image, generating category text lems based on textual descriptions of the candidate category names, and generating a plurality of contextual cue vectors using a cue generator. The contextual hint vector defines contextual information associated with an image classification task to be performed by the VLM. The method further includes generating a hint for each of the plurality of candidate categories by attaching the respective category text lemma to a context hint vector for each of the plurality of candidate categories, and generating and outputting a category prediction probability of the sample image based on the plurality of context hint vectors using the VLM.

Description

Technical Field

[0001] The present disclosure relates to prompt learning of vision-language models in computer vision systems and methods. Background Art

[0002] Computer vision systems such as image classification, image object detection, and image semantic segmentation techniques can use various models to predict, classify, detect, and label objects and regions in captured images. For example, a subset of data samples is used to train one or more machine learning models for image classification and labeling. Summary of the invention

[0003] A method for generating text-driven prompts and category prediction probabilities using a visual-language model (VLM) includes: receiving candidate category names associated with multiple candidate categories of an image, generating category text tokens based on text descriptions of the candidate category names, and using a prompt generator to generate multiple contextual prompt vectors. The contextual prompt vectors define contextual information associated with an image classification task to be performed by the VLM. The method further includes generating prompts for each of the multiple candidate categories by appending the corresponding category text token to the contextual prompt vector of each of the multiple candidate categories, and using the VLM to generate and output a category prediction probability for a sample image based on the multiple contextual prompt vectors.

[0004] In other features, the VLM is a contrastive language-image pre-training (CLIP) model. The method further includes receiving a text description at an interface. Receiving the text description includes providing a predefined text template and receiving the text description according to the predefined text template. The method further includes providing the text description to a large language model (LLM), and generating a text embedding using the large language model. The method further includes using a prompt generator model f θ To generate multiple contextual hint vectors. The method further includes mapping the text embedding to the multiple contextual hint vectors using a hint generator model. The method further includes aggregating multiple hint generator models f θ The corresponding parameters of the image encoder are obtained by collecting the corresponding parameters of the image encoder and outputting the collected prompt generator model based on the collected corresponding parameters. The VLM includes a text encoder and an image encoder.

[0005] A computing device configured to generate text-driven prompts and category prediction probabilities using a visual-language model (VLM), comprising a processing device configured to execute instructions stored in a memory to: receive candidate category names associated with multiple candidate categories of an image, generate category text tokens based on text descriptions of the candidate category names, generate multiple contextual prompt vectors using a prompt generator, the contextual prompt vectors defining contextual information associated with an image classification task to be performed by the VLM, generate prompts for each of the multiple candidate categories by appending corresponding category text tokens to the contextual prompt vector of each of the multiple candidate categories, and generate and output category prediction probabilities for a sample image based on the multiple contextual prompt vectors using the VLM.

[0006] In other features, the VLM is a contrastive language-image pre-training (CLIP) model. The interface is configured to receive a text description. The interface is configured to provide a predefined text template and receive a text description according to the predefined text template. The large language model (LLM) is configured to generate a text embedding. The prompt generator model f θ The VLM is configured to generate a plurality of contextual hint vectors. The hint generator model is configured to map text embeddings to the plurality of contextual hint vectors. The VLM includes a text encoder and an image encoder.

[0007] A computer-controlled machine comprises: at least one sensor configured to generate an input image; a control system configured to generate text-driven prompts and category prediction probabilities using a visual-language model (VLM), the control system configured to: receive candidate category names associated with multiple candidate categories of the input image, generate multiple contextual prompt vectors using a prompt generator, the contextual prompt vectors defining contextual information associated with an image classification task to be performed by the VLM, generate prompts for each of the multiple candidate categories by appending corresponding category text words to the contextual prompt vector of each of the multiple candidate categories, and generate and output category prediction probabilities for the input image based on the multiple contextual prompt vectors using the VLM; and an actuator configured to control the operation of the computer-controlled machine based on the category prediction probabilities.

[0008] In other features, the VLM is a contrastive language-image pre-training (CLIP) model. The computer-controlled machine further includes a prompt generator model f configured to generate a plurality of contextual prompt vectors. θ The prompt generator model is configured to map text embeddings to multiple contextual prompt vectors. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 A system for training a neural network according to the principles of the present disclosure is generally illustrated.

[0010] Figure 2 A computer-implemented method for training and utilizing a neural network in accordance with the principles of the present disclosure is generally illustrated.

[0011] Figure 3A An audio data tagging system according to the principles of the present disclosure is generally illustrated.

[0012] Figure 3B A portion of a data capture system in accordance with the principles of the present disclosure is generally illustrated.

[0013] Figure 3C An alternative audio data labeling system in accordance with the principles of the present disclosure is generally illustrated.

[0014] Figure 4A An example interface configured to receive a textual description of a target image classification problem in accordance with the principles of the present disclosure is generally illustrated.

[0015] Figure 4B An example functional block diagram of a text-to-classifier system in accordance with the principles of the present disclosure is generally illustrated.

[0016] Figure 4C The steps of an example category prediction method according to the principles of the present disclosure are generally illustrated.

[0017] Figure 4D The steps of an example model training method according to the principles of the present disclosure are generally illustrated.

[0018] Figure 5 Depicted is a schematic diagram of the interaction between a computer-controlled machine and a control system in accordance with the principles of the present disclosure.

[0019] Figure 6 Depicted is a device configured to control a vehicle according to the principles of the present disclosure. Figure 5 Schematic diagram of a control system of a vehicle, wherein the vehicle can be a partially autonomous vehicle, a fully autonomous vehicle, a partially autonomous robot, or a fully autonomous robot.

[0020] Figure 7 Depicted Figure 5 Schematic diagram of a control system configured to control a manufacturing machine, such as a punch cutter, a cutting machine, or a gun drill, of a manufacturing system (eg, part of a production line).

[0021] Figure 8 Depicted Figure 5 Schematic diagram of a control system configured to control a power tool, such as a power drill or a power screwdriver, having an at least partially autonomous mode.

[0022] Fig. 9Depicting a device configured to control an automated personal assistant Figure 5 Schematic diagram of the control system.

[0023] Fig.10 Depicted Figure 5 Schematic diagram of a control system configured to control a monitoring system, such as a controlled access system or a surveillance system.

[0024] Fig.11 Depicted Figure 5 Schematic diagram of a control system configured to control an imaging system, such as an MRI device, an x-ray imaging device, or an ultrasound device. DETAILED DESCRIPTION

[0025] Embodiments of the present disclosure are described herein. However, it is to be understood that the disclosed embodiments are merely examples, and other embodiments may take different and alternative forms. The figures are not necessarily to scale; some features may be magnified or minimized to show the details of a particular component. Therefore, the specific structural and functional details disclosed herein should not be interpreted as restrictive, but merely as a representative basis for teaching those skilled in the art to use the embodiments differently. As will be understood by those of ordinary skill in the art, the various features illustrated and described with reference to any one of the figures may be combined with the features illustrated in one or more other figures to produce embodiments that are not explicitly illustrated or described. The combination of illustrated features provides representative embodiments of typical applications. However, for a particular application or implementation, various combinations and modifications of features consistent with the teachings of the present disclosure may be required.

[0026] As used herein, "a", "an" and "the" refer to both the singular and the plural, unless the context clearly indicates otherwise. For example, a "processor" programmed to perform various functions refers to one processor programmed to perform each function, or more than one processor programmed together to perform each of the various functions.

[0027] A visual-language machine learning model for computer vision systems and methods is trained using a subset of data samples from a dataset. In some examples, the model is trained using data of labeled images. However, for zero-shot prediction tasks, such as zero-shot image classification, images are classified into different categories, and the model is not trained using data samples of labeled examples from the corresponding categories. For example, the model is trained using an image dataset and a textual description of the image category or the category of the object in the image.

[0028] In some examples, a base visual-language (VL) model or VLM (such as the Contrastive Language-Image Pretraining (CLIP) model) acquires comprehensive semantic knowledge by learning from a huge collection of image and text pairs, and can extract high-level features from high-dimensional inputs (e.g., images and language). Different base models focus on different data modalities and different tasks.

[0029] For example, the CLIP model can be used for zero-shot prediction tasks. For example, the CLIP model learns a text encoder E by training on a large number (e.g., 400 million) of image-text pairs. text and image encoder E image To predict the category y of a query image x, we can input a simple text prompt (e.g., “photos of [category]”) where the category tokens are replaced by the specific category names. The probability that image x belongs to category i is:

[0030] However, hand-crafted hint prefixes such as "photos of..." are not optimal in many cases.

[0031] In some examples, contextual optimization ("CoOp") techniques are used to implement cue learning for VMLs such as the CLIP model. For example, the CoOp technique replaces handcrafted cue prefixes, such as "photo of..." used to cue the CLIP text encoder, with a learnable text cue vector. The cue vector is learned from a small subset of labeled images from the target dataset. One text cue is learned independently for each category.

[0032] In other examples, a conditional CoOp ("CoCoOp") technique is used to generate an image-specific text hint vector conditioned on image features. The CoCoOp technique improves generalization to unseen image categories relative to the CoOp technique.

[0033] In other examples, the Multimodal Hint Learning (“MaPLe”) technique is used to learn hint vectors for both the text encoder and the image encoder. The MaPLe technique inserts learnable hint vectors into the intermediate transformer layers, while the CoOp and CoCoOp techniques only insert hint vectors into the input transformer layers.

[0034] These and other example techniques are configured to learn one model and a set of prompt vectors for one target classification problem. However, the underlying VL model of these techniques is used across many different classification problems, and learning a set of prompts for each problem can result in significant computational costs.

[0035] The prompt generation system and method according to the present disclosure uses a natural language description of a task to provide a customized image classifier for a target image classification task. For example, a prompt generator parameterized by a neural network (also referred to as a prompt translator or refiner) is used to convert a text description (e.g., user input) into a contextual prompt vector for a large VLM (e.g., a CLIP model). For example, a hand-crafted prompt prefix (e.g., "photo of...") can be replaced with a learnable contextual prompt vector (e.g., "[V_context][CLASS]"), which is provided to a text encoder. The contextual prompt vector converted from the textual description of the task provides contextual information about the task. By minimizing the loss of the CLIP model on some labeled images from a target dataset in a supervised manner, the contextual prompt vector can be learned for a target classification problem.

[0036] The large language model provides a powerful language interface for the VLM. According to the text-to-classifier (TTC) machine learning (ML) system disclosed herein, the large language model, the prompt generator and the VLM are composed. The prompt generator is configured to align the base model of the VLM for visual data and the language model. The VLM conditioned by the contextual prompt vector is convenient for predicting the category of the queried image. As used herein, "conditioning" refers to the process of providing additional information (e.g., text description, labeling, etc.) about the image to the VLM or other models before performing the computer vision tasks including classification, object detection and semantic segmentation. Therefore, "conditioning on" includes providing additional information to the model.

[0037] In this way, the systems and methods according to the present disclosure allow the image classifier for the target image classification task to be customized using a natural language description of the task, which is used to generate a contextual hint vector. The contextual hint vector converted from the text description can be generalized to different classification problems. Therefore, a model (e.g., a hint generator) learned / generated according to the principles of the present disclosure can be used to solve different classification problems. These systems and methods can be extended to applications beyond image classification, such as semantic segmentation and object detection.

[0038] Figure 1 An example system 100 for training a neural network (e.g., an ML model) is illustrated. System 100 may be configured to implement (and / or include circuitry configured to implement) the systems and methods of the present disclosure as described in more detail below. System 100 may include an input interface for accessing training data 102 for the neural network. For example, Figure 1As shown in , the input interface can be constituted by a data storage device interface 104, which can access the training data 102 from a data storage device 106. For example, the data storage device interface 104 can be a memory interface or a permanent storage device interface, such as a hard disk or SSD interface, but can also be a personal area network, local area network or wide area network interface, such as a Bluetooth, Zigbee or Wi-Fi interface or an Ethernet or fiber optic interface. The data storage device 106 can be an internal data storage device of the system 100, such as a hard disk drive or SSD, but can also be an external data storage device, such as a network accessible data storage device.

[0039] In some embodiments, data store 106 may further include a data representation 108 of an untrained version of the neural network, which data representation may be accessed by system 100 from data store 106. However, it will be appreciated that training data 102 and data representation 108 of the untrained neural network may also be accessed from different data stores, respectively, such as via different subsystems of data store interface 104. Each subsystem may be of a type as described above for data store interface 104.

[0040] In some embodiments, the data representation 108 of the untrained neural network may be generated internally by the system 100 based on the design parameters of the neural network, and thus may not be explicitly stored on the data storage device 106. The system 100 may also include a processor subsystem 110 that may be configured to provide an iterated function as a replacement for a layer stack of the neural network to be trained during operation of the system 100. Here, the corresponding layers of the replaced layer stack may have mutually shared weights and may receive the output of the previous layer as input, or, for the first layer of the layer stack, receive an initial activation and a portion of the input of the layer stack.

[0041] The processor subsystem 110 may further be configured to iteratively train the neural network using the training data 102. Here, the training iterations by the processor subsystem 110 may include a forward propagation portion and a backward propagation portion. The processor subsystem 110 may be configured to perform the forward propagation portion by determining, among other operations that define the forward propagation portion that may be performed, an equilibrium point of the iterative function at which the iterative function converges to a fixed point, and by providing the equilibrium point as a substitute for the output of the layer stack in the neural network, wherein determining the equilibrium point includes using a numerical root finding algorithm to find a root solution of the iterative function minus its input.

[0042] The system 100 may also include an output interface for outputting a data representation 112 of the trained neural network, which data may also be referred to as trained model data 112. Figure 1, the output interface may be comprised of a data storage device interface 104, which in these embodiments is an input / output ("IO") interface, via which trained model data 112 may be stored in a data storage device 106. For example, data representations 108 defining an "untrained" neural network may be at least partially replaced by data representations 112 of a trained neural network during or after training, as parameters of the neural network (e.g., weights, hyperparameters, and other types of parameters of the neural network) may be adapted to reflect training based on training data 102. This is also Figure 1 106 by reference numerals 108, 112, which refer to the same data record on the data store 106. In some embodiments, the data representation 112 may be stored separately from the data representation 108 defining the "untrained" neural network. In some embodiments, the output interface may be separate from the data store interface 104, but may generally be of the type described above for the data store interface 104.

[0043] Figure 2 A data annotation / augmentation system 200 is depicted, which is configured to implement a system for annotating and / or augmenting data (and / or includes circuitry configured to implement a system for annotating and / or augmenting data). The data annotation system 200 may include at least one computing system 202 configured to implement all or part of the systems and methods of the present disclosure explained in more detail below. The computing system 202 may include at least one processor 204, which may be operably connected to a memory unit 208. The processor 204 may include one or more integrated circuits that implement the functionality of a central processing unit (CPU) 206. The CPU 206 may be a commercially available processing unit that implements an instruction set such as one of the x86, ARM, Power, or MIPS instruction set families. The various components of the system 200 may be implemented with the same or different circuitry.

[0044] During operation, CPU 206 may execute stored program instructions retrieved from memory unit 208. The stored program instructions may include software that controls the operation of CPU 206 to perform the operations described herein. In some embodiments, processor 204 may be a system on a chip (SoC) that integrates the functionality of CPU 206, memory unit 208, network interfaces, and input / output interfaces into a single integrated device. Computing system 202 may implement an operating system for managing different aspects of operation.

[0045] Memory unit 208 may include volatile memory and nonvolatile memory for storing instructions and data. Nonvolatile memory may include solid-state memory, such as NAND flash memory, magnetic and optical storage media, or any other suitable data storage device that retains data when computing system 202 is disabled or loses power. Volatile memory may include static and dynamic random access memory (RAM) that stores program instructions and data. For example, memory unit 208 may store machine learning model 210 (e.g., in Figure 2 210), or an algorithm, a training data set 212 for the machine learning model 210, an original source data set 216, etc.

[0046] The computing system 202 may include a network interface device 222 that is configured to provide communications with external systems and devices. For example, the network interface device 222 may include a wired and / or wireless Ethernet interface defined by the Institute of Electrical and Electronics Engineers (IEEE) 802.11 family of standards. The network interface device 222 may include a cellular communication interface for communicating with a cellular network (e.g., 3G, 4G, 5G). The network interface device 222 may also be configured to provide a communication interface to an external network 224 or a cloud.

[0047] The external network 224 may be referred to as the World Wide Web or the Internet. The external network 224 may establish a standard communication protocol between computing devices. The external network 224 may allow information and data to be easily exchanged between computing devices and the network. One or more servers 230 may communicate with the external network 224.

[0048] The computing system 202 may include an input / output (I / O) interface 220, which may be configured to provide digital and / or analog input and output. The I / O interface 220 may include an additional serial interface (e.g., a universal serial bus (USB) interface) for communicating with external devices. The computing system 202 may include a human-machine interface (HMI) device 218, which may include any device that enables the system 200 to receive control inputs. Examples of input devices may include human-machine interface inputs, such as keyboards, mice, touch screens, voice input devices, and other similar devices. The computing system 202 may include a display device 232. The computing system 202 may include hardware and software for outputting graphics and text information to the display device 232. The display device 232 may include an electronic display screen, a projector, a printer, or other suitable device for displaying information to a user or operator. The computing system 202 may also be configured to allow interaction with a remote HMI and a remote display device via a network interface device 222.

[0049] System 200 may be implemented using one or more computing systems. Although the example depicts a single computing system 202 that implements all of the described features, it is intended that the various features and functions may be separated and implemented by multiple computing units that communicate with each other. The specific system architecture selected may depend on a variety of factors.

[0050] The system 200 can implement a machine learning model 210 configured to analyze a raw source data set 216. For example, the CPU 206 and / or other circuits can implement the machine learning model 210. The raw source data set 216 can include raw or unprocessed sensor data that can represent an input data set for a machine learning system. The raw source data set 216 can include video, video clips, images, audio, text-based information, and raw or partially processed sensor data (e.g., a radar map of an object). In some embodiments, the machine learning model 210 can be a deep learning or neural network algorithm designed to perform a predetermined function. For example, a neural network algorithm can be configured to identify an event or object in a video clip based on audio data.

[0051] The computer system 200 may store a training data set 212 for the machine learning model 210. The training data set 212 may represent a set of previously constructed data used to train the machine learning model 210. The training data set 212 may be used by the machine learning model 210 to learn weighting factors associated with a neural network algorithm. The training data set 212 may include a set of source data having corresponding achievements or results that the machine learning model 210 attempts to replicate via a learning process.

[0052] The machine learning model 210 can operate in a learning mode using the training data set 212 as input. The machine learning model 210 can be executed through multiple iterations using data from the training data set 212. With each iteration, the machine learning model 210 can update the internal weighting factors based on the results obtained. For example, the machine learning model 210 can compare the output results (e.g., annotations) with the results included in the training data set 212. Because the training data set 212 includes expected results, the machine learning model 210 can determine when the performance is acceptable. After the machine learning model 210 reaches a predetermined performance level (e.g., 100% consistent with the results associated with the training data set 212), the machine learning model 210 can be executed using data that is not in the training data set 212. The trained machine learning model 210 can be applied to a new data set to generate annotated data.

[0053] The machine learning model 210 may be configured to identify specific features in the raw source data 216. The raw source data 216 may include multiple instances or input data sets for which annotation results are desired (e.g., video streams or segments including audio data). By way of example only, the machine learning model 210 may be configured to identify objects or events in a video segment based on audio data and annotate these events. The machine learning model 210 may be programmed to process the raw source data 216 to identify the presence of specific features. The machine learning model 210 may be configured to identify features in the raw source data 216 as predetermined features. The raw source data 216 may be derived from a variety of sources. For example, the raw source data 216 may be actual input data collected by a machine learning system. The raw source data 216 may be machine generated for testing the system. As an example, the raw source data 216 may include raw video and / or audio data from a camera, audio data from a microphone, and the like.

[0054] In one example, the machine learning model 210 can process the raw source data 216 and output video and / or audio data including one or more indications of the identified events. The machine learning model 210 can generate a confidence level or factor for each generated output. For example, a confidence value exceeding a predetermined high confidence threshold can indicate that the machine learning model 210 is confident that the identified event (or feature) corresponds to a particular event. A confidence value less than a low confidence threshold can indicate that the machine learning model 210 has some uncertainty that a particular feature exists.

[0055] like Figure 3A and 3B , system 300 may include an image (e.g., video) capture device 302, an audio capture array 304, and a computing system 202. The system may receive video stream data associated with a data capture environment from the image capture device 302. The system 202 may be configured to perform video object detection to identify one or more objects in a corresponding image of the video stream data. The system 202 may receive audio stream data corresponding to at least a portion of the video stream data from the audio capture array 304. The audio capture array 304 may include one or more microphones 306 or other suitable audio capture devices. The systems and methods described herein may be configured to use output from at least a first machine learning model (e.g., such as machine learning model 210 or other suitable machine learning model configured to provide output including one or more object or event detection predictions) to label at least some objects of the video stream data and / or audio stream data.

[0056] The system 202 may calculate (e.g., using at least one probability-based function or other suitable technique or function) at least one offset value for at least a portion of the audio stream data corresponding to at least one tagged object of the video stream data based on at least one data capture characteristic. The system 202 may use the at least one offset value to synchronize at least a portion of the video stream data with the portion of the audio stream data corresponding to at least one tagged object of the video stream data. The at least one data capture characteristic may include one or more characteristics of at least one image capture device, one or more characteristics of at least one audio capture array, one or more characteristics corresponding to a position of at least one image capture device relative to at least one audio capture array, one or more characteristics corresponding to movement of an object in the video stream data, one or more other suitable data capture characteristics, or a combination thereof.

[0057] The system 202 may use one or more tags of the labeled categories or objects of the video stream data and at least one offset value to mark at least the portion of the audio stream data corresponding to at least one labeled category or object of the video stream data. Each corresponding tag may include an event type, an event start indicator, and an event end indicator. The system 202 may use at least some of the labeled portions of the audio stream data to generate training data. The system 202 may use the training data to train a second machine learning model. The system 202 may use the second machine learning model to detect one or more sounds associated with the audio data provided as input to the second machine learning model. The second machine learning model may include any suitable machine learning model and may be configured to perform any suitable functions, such as those described herein with respect to Figures 4-11.

[0058] In some embodiments, Figure 3C, the computing system 202 can be configured to tag audio data based on sensor data received from one or more sensors, such as those described herein or any other suitable sensor or sensor combination. The system 202 can receive audio stream data associated with the data capture environment from the audio capture array 354 or any suitable audio capture device (such as one or more of the microphones 306 or other suitable audio capture devices). It should be understood that the audio capture array 354 may include features similar to those of the audio capture array 304, and may include any suitable number of audio capture devices. The system 202 can receive sensor data associated with the data capture environment from at least one sensor (e.g., such as sensor 352) that is asynchronous with respect to the audio capture array 354. The sensor 354 may include at least one of an induction coil, a radar sensor, a lidar sensor, a sonar sensor, an image capture device, any other suitable sensor, or a combination thereof. The audio capture array 354 can be positioned away from the sensor 354, positioned close to the sensor 354, or positioned in any suitable relationship with the sensor 354.

[0059] System 202 may use output from at least a first machine learning model (e.g., machine learning model 210 or other suitable machine learning model) to identify at least some events in sensor data. Machine learning model 210 may be configured to provide output including one or more event detection predictions based on sensor data. System 202 may synchronize at least a portion of sensor data associated with a portion of audio stream data corresponding to at least one event of sensor data. System 202 may use one or more tags extracted for corresponding events of sensor data values ​​to mark at least the portion of audio stream data corresponding to at least one event of sensor data. Each corresponding tag may include an event type, an event start indicator, and an event end indicator. System 202 may use at least some of the marked portions of audio stream data to generate training data. System 202 may use training data to train a second machine learning model. System 202 may use a second machine learning model to detect one or more sounds associated with audio data provided as input to the second machine learning model. The second machine learning model may include any suitable machine learning model and may be configured to perform any suitable functions, such as those described herein with respect to FIGS. 4-11.

[0060] Any of the systems described above and / or in more detail below may implement the prompt generation systems and methods of the present disclosure, including the prompt generator described in more detail below.

[0061] The TTC system according to the present disclosure includes a large language model, a visual-language model (VLM), and a prompt generator (e.g., a prompt generator model) configured to align the VLM and the large language model. The large language model LLM (e.g., a transformer-based model such as a Bidirectional Encoder Representations from Transformers (BERT) model) embeds a variable-length raw text description of a classification problem. context As text embeddings (e.g., categorical text tokens). Text embeddings extract and summarize contextual information from raw text descriptions (e.g., raw text descriptions entered by users). Text embeddings are provided as input to the prompt generator model.

[0062] Prompt Generator Model f θ Variable-length text embeddings are mapped to a set of fixed-length contextual hint vectors. The hint generator model can include both a cross-attention layer and a subsequent self-attention layer.

[0063] VLM (e.g., with a text encoder E text and image encoder E image The CLIP model (of the type described in

[15] ) predicts the category of a query image based on a contextual cue vector. The text encoder is a transformer model that encodes text tokens into text features. The image encoder is a visual transformer model that encodes an image into image features. At least one of the text encoder and the image encoder is conditioned on (e.g., provided with) the contextual cue vector.

[0064] The parameters of the large language model and the visual-language model (e.g., the text encoder and image encoder of the CLIP model) can be pre-trained and kept fixed in the TTC system. Therefore, in some examples, only the parameters of the prompt generator model are learned using a small set of labeled images. A set of image-text pairs can be used for model training. A given set of image-text pairs (e.g., a mini-batch) can include n image-text pairs from k different image categories. Using k image category names c0...c k Generate text descriptions for classification problems using predefined text templates for training context For example, the predefined text template can be “We have images from the following categories: [category name]”, where [category name] is the i-th image category name c i and [k] by replacing the i in the text description t context is fed to the large language model LLM and then to the prompt generator model f θ , the contextual hint vector is obtained for the classification problem of the given set of image-text pairs according to the following formula: V text , V image =fθ (LLM(t context )) (1), Conditioned on the contextual cue vector, the CLIP model predicts the probability: where cos() corresponds to cosine similarity.

[0065] The parameters θ of the hint translator model are learned by minimizing the cross entropy between the model predictions and the ground truth tokens over the available image-text pairs in each mini-batch.

[0066] For inference (e.g., predicting the class of a new image), for the zero-shot image classification task, the TTC system receives as input a textual description of the image classification problem to be performed. The textual description is converted into a contextual cue vector by a large language model and a cue generator model, as shown in Equation 1 above. The CLIP model conditioned on the contextual cue vector then predicts the image class as shown in Equation 2 above.

[0067] As described herein, appropriate hints provide helpful contextual information for the target classification problem. However, different image classification tasks require different contexts. For example, classifying different pets relies on different characteristic features than classifying different flowers, and hints learned for pet image classification are not optimal for flower image classification. Equations 1 and 2 are configured to generate hint vectors based on corresponding text descriptions of the target image classification problem, and the hint generator model is configured to convert a set of candidate class names into a contextual hint vector.

[0068] In some examples, the text description can be automatically generated from metadata associated with one or more images. For example, the text description can be "We have an image from a camera of product A at machine B in factory C," where details such as machine B and factory C can be obtained from the metadata associated with the image.

[0069] In some examples, in addition to the contextual hint vector V text In addition, we can use a hint vector V that depends on the category name i =f θ (LLM(c i )). CLIP text encoder can be used by text and V i and the category name c in the text encoder input layer i Cascade to V text and V iBoth are conditional. Contextual cue vectors and cue vectors that depend on category names can be inserted into other transformation layers (e.g., as in a multi-model cue learning system).

[0070] The above TTC system as described above can be used for tasks other than image classification. For example, for semantic segmentation, the TTC system may include a large language model, a visual model configured for semantic segmentation, and a prompt generator configured to align the large language model and the visual model. On the other hand, for image classification or object detection, the TTC system may include a large language model, a visual model configured for image classification or object detection, and a prompt generator configured to align the large language model and the visual model.

[0071] Figure 4A An example interface 400 is illustrated that is configured to receive (eg, from a user) a text description of a target image classification problem. For example, the text description indicates two categories of images (eg, images of fully coated surfaces and images of non-fully coated surfaces).

[0072] Figure 4B An example functional block diagram of a text-to-classifier system 404 trained according to the principles of the present disclosure is generally illustrated.For example, one or more processors or processing devices are configured to execute instructions to implement the system 404, such as one or more processors of the systems described herein.

[0073] Database 406 is populated with data 408 associated with one or more images, such as an image corresponding to image file 410. In this example, the data may include the part shown in the image, the factory and station associated with the part, the task associated with the part (e.g., surface coating), and the category associated with the part shown in the image (e.g., fully covered / coated surface and not fully covered / coated surface).

[0074] System 404 provides a text template 412 (i.e., a predefined text template as described above) that is configured to receive a text description 414 of a target image classification problem to be performed by the system. For example, system 404 (e.g., via a method such as Figure 4A The text description 414 is provided to a large language model (LLM) 416.

[0075] Large language model 416 embeddings for variable length raw text descriptions of classification problems context As a text embedding, the text embedding is provided to a prompt generator model f configured to implement θ Hint generator 418. Hint generator 418 maps variable length text embeddings to a set of fixed length contextual hint vectors 420.

[0076] The contextual cue vector 420 and the category (shown at 422) are provided to a VLM 426 (eg, including a text encoder 424 and an image encoder 428 (E image )'s CLIP model)'s text encoder 424 (E text ). The text encoder 424 encodes the text tokens into text features. On the other hand, the image encoder 428 receives the image 430 (e.g., an image corresponding to the image file 410) and encodes the image 430 into image features. In this way, the hint generator 418 is configured to align the LLM 416 and the VLM 426. The VLM 426 conditioned on the context hint vector 420 facilitates predicting the category of the queried image. Therefore, the VLM 426 is configured to predict the category of the query image (e.g., image 430) based on the context hint vector 420.

[0077] Figure 4C Steps of a category prediction method 440 implemented by the TTC system 404 and the VLM 426 according to the principles of the present disclosure are illustrated. For example, one or more processors or processing devices are configured to execute instructions to implement the method 440, such as one or more processors of the system described herein.

[0078] Assuming that the classification problem involves n candidate categories, a set of candidate category tokens is:

[0079] Hint generator model f with learnable parameters θ θ The group of category tokens T is converted into m soft-hint prefix vectors as:

[0080] The hint generator model is parameterized by a lightweight criss-cross attention module followed by a multi-layer perceptron (MLP). Therefore, the soft hint vector for the jth class corresponds to: t j ={v1,v2,...,v m , c j}.

[0081] Given an image x, an image encoder is used to extract visual embeddings, and a text encoder is used to extract text embeddings. The prediction probability is calculated as follows: This is a variation of equation 2 above.

[0082] Thus, at 444, method 440 collects / receives test data xtest (e.g., sample image) and candidate category name tokens T test At 448, method 440 performs the following steps according to V i =f θ ((T test )) based on the test data and candidate category name tokens (e.g., using a prompt generator model f θ ) contextual cue vector v, where V i Corresponding to the lemma soft hint prefix vectors V1, V2, ... V m At 452, method 440 performs the following steps according to t j = {v1, v2, ..., v m , c j} to obtain the soft hint vector for each category.

[0083] At 456 , method 440 calculates and outputs the class prediction probability of the sample image according to Equation 3.

[0084] Figure 4D The steps of an example model training method 460 implemented by the TTC system 404 and the VLM 426 according to the principles of the present disclosure are generally illustrated. For example, one or more processors or processing devices are configured to execute instructions to implement the method 440, such as one or more processors of the system described herein.

[0085] For example, a generalized federated learning setting may include a central server and N different remote clients indexed as N = [1, 2, ..., N]. The i-th client has a local dataset with labeled images (xi, yi) ~ Di and the n-th available class name. All clients have disjoint data distributions (i.e., both images and class names at different clients are non-overlapping).

[0086] The goal is to learn a generalized cue generator model f with learnable parameters θ θ . Tip generator model f θ is configured to convert a set of candidate class names into a contextual cue vector for the target image classification problem. Each client locally updates the model parameters θ by minimizing the loss of the CLIP model on the corresponding private data Di. The model parameters θ calculated to minimize the loss at a given client correspond to:

[0087] The model parameters are aggregated (eg, at a central server) as follows:

[0088] Therefore, at 464, method 460 collects / receives a local training dataset Di and available category name tokens T i At 468, method 460 performs the following steps according to V i =f θ ((T i )) based on the training dataset and available category name tokens (e.g., using a prompt generator model f θ ) context hint vector v. At 472, method 460 calculates the context hint vector v according to t j = {v1, v2, .., v m , c j}Get the soft hint vector for each category.

[0089] At 476 , method 460 updates and stores and / or outputs model parameters θ and the aggregated model parameters according to Equations 4 and 5.

[0090] Figure 5-11 Depicted are example systems and devices that may implement model training and / or category prediction systems and methods according to the present disclosure. Figure 5 A schematic diagram of the interaction between a computer-controlled machine 500 and a control system 502 is depicted. The computer-controlled machine 500 includes an actuator 504 and a sensor 506. The actuator 504 may include one or more actuators, and the sensor 506 may include one or more sensors. The sensor 506 is configured to sense a condition of the computer-controlled machine 500. The sensor 506 may be configured to encode the sensed condition into a sensor signal 508 and transmit the sensor signal 508 to the control system 502. Non-limiting examples of the sensor 506 include a video sensor, a radar sensor, a lidar sensor, an ultrasonic sensor, and a motion sensor. In some embodiments, the sensor 506 is an optical sensor configured to sense an optical image of an environment near the computer-controlled machine 500.

[0091] The control system 502 is configured to receive sensor signals 508 from the computer controlled machine 500. As explained below, the control system 502 may also be configured to calculate actuator control commands 510 depending on the sensor signals and transmit the actuator control commands 510 to the actuators 504 of the computer controlled machine 500.

[0092] like Figure 5As shown in , the control system 502 includes a receiving unit 512. The receiving unit 512 may be configured to receive a sensor signal 508 from the sensor 506 and convert the sensor signal 508 into an input signal x. In an alternative embodiment, the sensor signal 508 is directly received as an input signal x without the receiving unit 512. Each input signal x may be a portion of each sensor signal 508. The receiving unit 512 may be configured to process each sensor signal 508 to generate each input signal x. The input signal x may include data corresponding to an image recorded by the sensor 506.

[0093] The control system 502 includes a classifier 514. The classifier 514 can be configured to classify the input signal x into one or more labels using a machine learning (ML) algorithm (such as a neural network). For example, the classifier 514 corresponds to the above-mentioned classifier 408. The classifier 514 is configured to be parameterized by parameters, such as those described above (e.g., parameter θ). The parameter θ can be stored in and provided by the non-volatile storage device 516. The classifier 514 is configured to determine the output signal y from the input signal x. Each output signal y includes information that assigns one or more labels to each input signal x. The classifier 514 can transmit the output signal y to a conversion unit 518. The conversion unit 518 is configured to convert the output signal y into an actuator control command 510. The control system 502 is configured to transmit the actuator control command 510 to the actuator 504, and the actuator 504 is configured to actuate the computer-controlled machine 500 in response to the actuator control command 510. In some embodiments, actuator 504 is configured to actuate computer-controlled machine 500 directly based on output signal y.

[0094] When the actuator control command 510 is received by the actuator 504, the actuator 504 is configured to perform an action corresponding to the associated actuator control command 510. The actuator 504 may include control logic configured to convert the actuator control command 510 into a second actuator control command used to control the actuator 504. In one or more embodiments, the actuator control command 510 may be used to control a display instead of or in addition to the actuator.

[0095] In some embodiments, the control system 502 includes the sensor 506 instead of or in addition to the computer-controlled machine 500 including the sensor 506. The control system 502 may also include the actuator 504 instead of or in addition to the computer-controlled machine 500 including the actuator 504.

[0096] like Figure 5As shown in , the control system 502 also includes a processor 520 and a memory 522. The processor 520 may include one or more processors. The memory 522 may include one or more memory devices. The classifier 514 (e.g., ML algorithm) of one or more embodiments may be implemented by the control system 502, which includes a non-volatile storage device 516, a processor 520, and a memory 522.

[0097] The non-volatile storage 516 may include one or more permanent data storage devices, such as a hard drive, an optical drive, a tape drive, a non-volatile solid-state device, a cloud storage device, or any other device capable of permanently storing information. The processor 520 may include one or more devices selected from a high-performance computing (HPC) system, including a high-performance core, a microprocessor, a microcontroller, a digital signal processor, a microcomputer, a central processing unit, a field programmable gate array, a programmable logic device, a state machine, a logic circuit, an analog circuit, a digital circuit, or any other device that manipulates signals (analog or digital) based on computer-executable instructions residing in the memory 522. The memory 522 may include a single memory device or multiple memory devices, including but not limited to random access memory (RAM), volatile memory, non-volatile memory, static random access memory (SRAM), dynamic random access memory (DRAM), flash memory, cache memory, or any other device capable of storing information.

[0098] The processor 520 may be configured to read into the memory 522 and execute computer executable instructions residing in the non-volatile storage device 516 and specifically implementing one or more abnormality monitoring methods of one or more embodiments. The non-volatile storage device 516 may include one or more operating systems and applications. The non-volatile storage device 516 may store programs compiled and / or interpreted from computer programs created using a variety of programming languages ​​and / or technologies, including but not limited to and alone or in combination Java, C, C++, C#, Objective C, Fortran, Pascal, Java Script, Python, Perl, and PL / SQL.

[0099] When executed by processor 520, the computer executable instructions of non-volatile storage device 516 may cause control system 502 to implement one or more of the anomaly monitoring methods disclosed herein. Non-volatile storage device 516 may also include data supporting the functions, features, and processes of one or more embodiments described herein.

[0100] The program code of the specific implementation algorithm and / or method described herein can be distributed individually or jointly as a program product in a variety of different forms. The program code can be distributed using a computer-readable storage medium having a computer-readable program instruction thereon, for causing the processor to perform various aspects of one or more embodiments. The inherently non-temporary computer-readable storage medium may include volatile and non-volatile and removable and non-removable tangible media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program modules or other data. The computer-readable storage medium may also include RAM, ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other solid-state storage technology, portable compact disc read-only memory (CD-ROM) or other optical storage device, cassette, tape, disk storage device or other magnetic storage device, or any other medium that can be used to store desired information and can be read by a computer. The computer-readable program instruction can be downloaded from the computer-readable storage medium to a computer, another type of programmable data processing device or another device, or downloaded to an external computer or external storage device via a network.

[0101] Computer-readable program instructions stored in a computer-readable medium can be used to instruct a computer, other types of programmable data processing devices, or other devices to operate in a particular manner so that the instructions stored in the computer-readable medium produce an article of manufacture including instructions for implementing the functions, actions, and / or operations specified in the flowchart or chart. In certain alternative embodiments, consistent with one or more embodiments, the functions, actions, and / or operations specified in the flowchart and chart can be reordered, processed serially, and / or processed concurrently. In addition, any flowchart and / or chart may include more or fewer nodes or blocks than those illustrated consistently with one or more embodiments.

[0102] The processes, methods or algorithms may be embodied in whole or in part using appropriate hardware components, such as application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), state machines, controllers or other hardware components or devices, or a combination of hardware, software and firmware components.

[0103] Figure 6A schematic diagram of a control system 502 configured to control a vehicle 600, which may be an at least partially autonomous vehicle or an at least partially autonomous robot, is depicted. The vehicle 600 includes an actuator 504 and a sensor 506. The sensor 506 may include one or more video sensors, cameras, radar sensors, ultrasonic sensors, lidar sensors, and / or location sensors (e.g., GPS). One or more of the one or more specific sensors may be integrated into the vehicle 600. Alternatively or in addition to the one or more specific sensors identified above, the sensor 506 may include a software module that is configured to determine the state of the actuator 504 when executed. A non-limiting example of a software module includes a weather information software module that is configured to determine the current or future state of the weather near the vehicle 600 or other location.

[0104] The classifier 514 of the control system 502 of the vehicle 600 may be configured to detect an object in proximity to the vehicle 600 depending on the input signal x. In such an embodiment, the output signal y may include information characterizing the proximity of the object to the vehicle 600. The actuator control command 510 may be determined based on this information. The actuator control command 510 may be used to avoid a collision with the detected object.

[0105] In some embodiments, the vehicle 600 is an at least partially autonomous vehicle, and the actuator 504 may be embodied in the brakes, propulsion system, engine, drive train, or steering of the vehicle 600. Actuator control commands 510 may be determined to control the actuator 504 so that the vehicle 600 avoids colliding with a detected object. The detected objects may also be classified according to what the classifier 514 thinks they are most likely to be, such as pedestrians or trees. The actuator control commands 510 may be determined depending on the classification. In the event of a possible hostile attack, the above system may be further trained to better detect objects or identify changes in lighting conditions or angles of sensors or cameras on the vehicle 600.

[0106] In some embodiments where the vehicle 600 is an at least partially autonomous robot, the vehicle 600 may be a mobile robot configured to perform one or more functions, such as flying, swimming, diving, and walking. The mobile robot may be an at least partially autonomous lawn mower or an at least partially autonomous cleaning robot. In such embodiments, actuator control commands 510 may be determined such that a propulsion unit, a steering unit, and / or a braking unit of the mobile robot may be controlled such that the mobile robot may avoid colliding with an identified object.

[0107] In some embodiments, vehicle 600 is an at least partially autonomous robot in the form of a gardening robot. In such embodiments, vehicle 600 may use an optical sensor as sensor 506 to determine the state of plants in the environment near vehicle 600. Actuator 504 may be a nozzle configured to spray a chemical. Depending on the identified type of plant and / or the identified state, actuator control command 510 may be determined to cause actuator 504 to spray an appropriate amount of an appropriate chemical to the plant.

[0108] The vehicle 600 may be an at least partially autonomous robot in the form of a household appliance. Non-limiting examples of household appliances include a washing machine, a stove, an oven, a microwave, or a dishwasher. In such a vehicle 600, the sensor 506 may be an optical sensor configured to detect the state of an object to be processed by the household appliance. For example, in the case where the household appliance is a washing machine, the sensor 506 may detect the state of the laundry in the washing machine. The actuator control command 510 may be determined based on the detected state of the laundry.

[0109] Figure 7 A schematic diagram of a control system 502 is depicted, the control system 502 being configured to control a system 700 (e.g., a manufacturing machine) of a manufacturing system 702, such as a punch cutter, a cutter, or a gun drill, for example, that is part of a production line. The control system 502 may be configured to control an actuator 504, which is configured to control the system 700 (e.g., a manufacturing machine).

[0110] The sensor 506 of the system 700 (e.g., a manufacturing machine) may be an optical sensor configured to capture one or more characteristics of the manufactured product 704. The classifier 514 may be configured to determine a state of the manufactured product 704 based on the one or more captured characteristics. The actuator 504 may be configured to control the system 700 (e.g., a manufacturing machine) for subsequent manufacturing steps of the manufactured product 704 depending on the determined state of the manufactured product 704. The actuator 504 may be configured to control the function of the system 700 (e.g., a manufacturing machine) on the subsequent manufactured product 706 of the system 700 (e.g., a manufacturing machine) depending on the determined state of the manufactured product 704.

[0111] Figure 8 A schematic diagram of a control system 502 configured to control a power tool 800 (eg, an electric drill or an electronic screwdriver) having at least a partially autonomous mode is depicted. The control system 502 may be configured to control an actuator 504 configured to control the power tool 800 .

[0112] The sensor 506 of the power tool 800 may be an optical sensor configured to capture one or more characteristics of the working surface 802 and / or the fastener 804 driven into the working surface 802. The classifier 514 may be configured to determine the state of the working surface 802 and / or the fastener 804 relative to the working surface 802 based on the one or more captured characteristics. The state may be that the fastener 804 is flush with the working surface 802. Alternatively, the state may be the hardness of the working surface 802. The actuator 504 may be configured to control the power tool 800 so that the driving function of the power tool 800 is adjusted depending on the determined state of the fastener 804 relative to the working surface 802 or the one or more characteristics of the captured working surface 802. For example, if the state of the fastener 804 is flush with the working surface 802, the actuator 504 may interrupt the driving function. As another non-limiting example, the actuator 504 may apply additional or less torque depending on the hardness of the working surface 802.

[0113] Fig. 9 Depicted is a schematic diagram of a control system 502 configured to control an automated personal assistant 900. The control system 502 may be configured to control an actuator 504, which is configured to control the automated personal assistant 900. The automated personal assistant 900 may be configured to control a household appliance, such as a washing machine, stove, oven, microwave, or dishwasher.

[0114] Sensor 506 may be an optical sensor and / or an audio sensor. The optical sensor may be configured to receive a video image of gesture 904 of user 902. The audio sensor may be configured to receive a voice command of user 902.

[0115] The control system 502 of the automated personal assistant 900 may be configured to determine an actuator control command 510 configured to control the system 502. The control system 502 may be configured to determine the actuator control command 510 based on the sensor signal 508 of the sensor 506. The automated personal assistant 900 is configured to transmit the sensor signal 508 to the control system 502. The classifier 514 of the control system 502 may be configured to execute a gesture recognition algorithm to identify the gesture 904 made by the user 902, thereby determining the actuator control command 510, and transmitting the actuator control command 510 to the actuator 504. The classifier 514 may be configured to retrieve information from the non-volatile storage device in response to the gesture 904, and output the retrieved information in a form suitable for receipt by the user 902.

[0116] Fig.10A schematic diagram of a control system 502 configured to control a monitoring system 1000 is depicted. The monitoring system 1000 can be configured to physically control access through a door 1002. A sensor 506 can be configured to detect scenes that are relevant in deciding whether to grant access. The sensor 506 can be an optical sensor configured to generate and transmit image and / or video data. Such data can be used by the control system 502 to detect faces.

[0117] The classifier 514 of the control system 502 of the monitoring system 1000 can be configured to interpret the image and / or video data by matching the identities of known persons stored in the non-volatile storage device 516 to determine the identity of the person. The classifier 514 can be configured to generate an actuator control command 510 in response to the interpretation of the image and / or video data. The control system 502 is configured to transmit the actuator control command 510 to the actuator 504. In this embodiment, the actuator 504 can be configured to lock or unlock the door 1002 in response to the actuator control command 510. In some embodiments, non-physical logical access control is also possible.

[0118] The monitoring system 1000 may also be a surveillance system. In such an embodiment, the sensor 506 may be an optical sensor configured to detect a scene under surveillance, and the control system 502 is configured to control the display 1004. The classifier 514 is configured to determine the classification of the scene, such as whether the scene detected by the sensor 506 is suspicious. The control system 502 is configured to transmit an actuator control command 510 to the display 1004 in response to the classification. The display 1004 may be configured to adjust the displayed content in response to the actuator control command 510. For example, the display 1004 may highlight an object that is considered suspicious by the classifier 514. By utilizing embodiments of the disclosed system, the monitoring system can predict objects that appear at a specific time in the future.

[0119] Fig.11 A schematic diagram of a control system 502 is depicted, the control system 502 being configured to control an imaging system 1100, such as an MRI device, an x-ray imaging device, or an ultrasound device. The sensor 506 may be, for example, an imaging sensor. The classifier 514 may be configured to determine a classification of all or part of a sensed image. The classifier 514 may be configured to determine or select an actuator control command 510 in response to a classification obtained by a trained neural network. For example, the classifier 514 may interpret an area of ​​a sensed image as a potential anomaly. In this case, the actuator control command 510 may be determined or selected to cause the display 1102 to display the imaging and highlight the potential abnormal area.

[0120] Although exemplary embodiments are described above, these embodiments are not intended to describe all possible forms contained by the claims. The words used in the specification are words of description rather than limitation, and it is understood that various changes can be made without departing from the spirit and scope of the present disclosure. As previously described, the features of the various embodiments can be combined to form more embodiments of the present invention that may not be explicitly described or illustrated. Although various embodiments may have been described as providing advantages in one or more desired characteristics or being preferred over other embodiments or prior art implementations, it is recognized by those of ordinary skill in the art that one or more features or characteristics can be compromised to achieve the desired overall system properties, depending on the specific application and implementation. These properties may include, but are not limited to, cost, strength, durability, life cycle cost, marketability, appearance, packaging, size, applicability, weight, manufacturability, ease of assembly, etc. Therefore, in terms of one or more characteristics, to the extent that any embodiment is described as less desirable than other embodiments or prior art implementations, these embodiments are not outside the scope of the present disclosure and may be ideal for specific applications.

Claims

1. A method for generating text-driven prompts and category prediction probabilities using a visual-language model (VLM), the method comprising: receiving candidate category names associated with a plurality of candidate categories for an image; Generate category text tokens based on the text description of the candidate category name; generating, using a hint generator, a plurality of context hint vectors, wherein the context hint vectors define context information associated with an image classification task to be performed by the VLM; generating a hint for each of the plurality of candidate categories by appending a corresponding category text token to a context hint vector for each of the plurality of candidate categories; as well as The VLM is used to generate and output the category prediction probability of the sample image based on the multiple context clue vectors.

2. The method of claim 1, wherein the VLM is a contrastive language-image pre-training (CLIP) model. The method of claim 1 , further comprising receiving a text description at an interface. The method according to claim 3 , wherein receiving the text description comprises providing a predefined text template and receiving the text description according to the predefined text template.

5. The method of claim 1 , further comprising providing the text description to a large language model (LLM), and generating a text embedding using the large language model.

6. The method of claim 1, further comprising using a prompt generator model f θ to generate the plurality of contextual hint vectors.

7. The method of claim 6, further comprising mapping text embeddings to the plurality of contextual hint vectors using a hint generator model.

8. The method of claim 6, further comprising aggregating a plurality of prompt generator models f θ and outputting the aggregated prompt generator model based on the aggregated corresponding parameters.

9. A computing device configured to generate text-driven prompts and category prediction probabilities using a visual-language model (VLM), the computing device comprising a processing device configured to execute instructions stored in a memory to: receive candidate category names associated with a plurality of candidate categories for an image; Generate category text tokens based on the text description of the candidate category name; generating, using a hint generator, a plurality of context hint vectors, wherein the context hint vectors define context information associated with an image classification task to be performed by the VLM; generating a hint for each of the plurality of candidate categories by appending a corresponding category text token to a context hint vector for each of the plurality of candidate categories; as well as The VLM is used to generate and output the category prediction probability of the sample image based on the multiple context clue vectors.

10. The computing device of claim 9, wherein the VLM is a contrastive language-image pre-training (CLIP) model.

11. The computing device of claim 9, further comprising an interface configured to receive the text description.

12. The computing device of claim 11, wherein the interface is configured to provide a predefined text template, and to receive the text description according to the predefined text template.

13. The computing device of claim 9, further comprising a large language model (LLM) configured to generate text embeddings.

14. The computing device of claim 9, further comprising a hint generator model f configured to generate the plurality of context hint vectors. θ .

15. The computing device of claim 14, wherein the prompt generator model is configured to map text embeddings to the plurality of contextual prompt vectors.

16. The computing device of claim 9, wherein the VLM comprises a text encoder and an image encoder.

17. A computer-controlled machine comprising: at least one sensor configured to generate an input image; A control system configured to generate text-driven prompts and category prediction probabilities using a vision-language model (VLM), the control system configured to: receiving candidate category names associated with a plurality of candidate categories for an input image, generating a plurality of contextual hint vectors using a hint generator, wherein the contextual hint vectors define contextual information associated with an image classification task to be performed by the VLM, generating a hint for each of the plurality of candidate categories by appending a corresponding category text token to a context hint vector for each of the plurality of candidate categories, and Generate and output a class prediction probability of the input image based on the plurality of contextual cue vectors using the VLM; as well as An actuator is configured to control operation of a computer-controlled machine based on the class prediction probabilities.

18. A computer controlled machine according to claim 17, wherein the VLM is a contrastive language-image pre-training (CLIP) model.

19. The computer controlled machine of claim 17, further comprising a hint generator model configured to generate the plurality of context hint vectors. θ .

20. The computer-controlled machine of claim 19, wherein the cue generator model is configured to map text embeddings to the plurality of contextual cue vectors.

Images