Classification by vision-language model with optimized text embedding

Abstract

A technique is discussed herein that facilitates optimization of a Vision-Language (VL)-based classifier with text embedding. The technique includes adjusting a visual-language model (VLM)-based classifier using a pre-trained image encoder of the VLM for image embedding of pre-classified images and using a pre-trained text encoder of the VLM for text embedding of a set of different text sentences. The technique further includes determining an optimized set of different text sentences in the superset of text sentences. The optimized set of the different text sentences has the minimum classification loss when the classifier based on the VL classifies the pre-classified image.

Description

[0001] introduce

[0002] Visual-Language Models (VLMs) are computerized models that combine computer vision and natural language processing techniques to generate content related to visual data. VLMs typically use a visual encoder to extract features from visual input and a language decoder to generate a text description or response. These models are trained on paired visual and textual data. These models can perform tasks such as image captioning and visual question answering. They facilitate text understanding and generation based on visual content, bridging the gap between visual and textual modalities. Summary of the invention

[0003] According to one embodiment, a method for facilitating the optimization of a visual-language (VL) based classifier using text embeddings. The method includes using a pre-trained image encoder of a visual-language model (VLM) for image embedding of pre-classified images and using a pre-trained text encoder of the VLM for text embedding of a set of different text sentences to adjust the VL based classifier. The method also determines an optimized set of different text sentences in a superset of text sentences. The optimized set of different text sentences has the minimum classification loss of the VL based classifier when classifying the pre-classified images. The method also includes providing an optimized set of different text sentences for the VL based classifier to classify unclassified images.

[0004] In this embodiment of the method, wherein the adjusting includes generating an image embedding from an image encoding of the pre-classified image using a pre-trained image encoder, and generating a text embedding from a text encoding of a collection of different text sentences using a pre-trained text encoder. The adjusting also includes classifying the pre-classified labeled image into one or more of a plurality of classifications based on a correlation between the image embedding and the text embedding by a VL-based classifier, and determining a classification loss for the pre-classified image based at least in part on the collection of text sentences.

[0005] For this embodiment of the method, the adjustment includes image encoding the pre-classified image using a pre-trained image encoder and generating an image embedding vector (x) from the output of the image encoding. The adjustment also includes text encoding a set of text sentences using a pre-trained text encoder, and generating a text embedding vector matrix (W) from the output of the text encoding. The adjustment also includes correlating the image embedding vector and the text embedding vector matrix to produce a projection matrix (Wx) and classifying the pre-classified labeled image into one or more of a plurality of classifications based on the projection matrix by a VL-based classifier. The adjustment also includes determining the classification loss of the pre-classified image based at least in part on the set of text sentences.

[0006] For this embodiment of the method, wherein the determining includes iteratively repeating the adjustment of the VL-based classifier, wherein each iteration includes supplying another set of different text sentences in a superset of text sentences. Each set of different text sentences includes at least one iteration-unique text sentence in the superset. The determining also includes: for each iteration, tracking the classification loss associated with the set of different text sentences for each iteration. The determining also includes identifying an optimized set of different text sentences in the superset, the optimized set of different text sentences being associated with a minimum classification loss for the VL-based classifier when classifying a pre-classified image.

[0007] For this embodiment of the method, wherein the determining includes employing an automatic text sentence generator.

[0008] In this embodiment of the method, the automatic text sentence generator is selected from the group consisting of: a language model (LM), a large language model (LLM), a Markov model (MM), a recursive neural network (RNN), a transformer, a template-based generator, a rule-based generator, a reinforcement learning (RL)-based generator, and a random word generator.

[0009] In this embodiment of the method, determining the classification loss includes calculating a set of cross entropy losses for different text sentences in the superset of text sentences.

[0010] Another embodiment of the method includes determining an optimal bias ("θ") for the VL-based classifier using an optimized set of different text sentences, wherein the optimal bias is associated with a minimum classification loss for the VL-based classifier when classifying the pre-classified image.

[0011] According to yet another embodiment, a non-transitory machine-readable storage medium is encoded with instructions executable by one or more processors, which when executed direct the one or more processors to perform operations of a VL-based classifier generated by the embodiment of the method described above. The VL-based classifier uses an optimized set of different text sentences for the VL-based classifier to classify unclassified images.

[0012] According to yet another embodiment, a method for facilitating optimization of a visual-language (VL) based classifier using text embeddings includes using a pre-trained image encoder of a visual-language model (VLM) for image embedding of pre-classified images and using a pre-trained text encoder of the VLM for text embedding of a set of different text sentences to adjust the VL based classifier. The method also determines an optimized set of different text sentences in a superset of text sentences. The optimized set of different text sentences has the minimum classification loss of the VL based classifier when classifying the pre-classified images. The determination includes using an automatic text sentence generator. The method also includes providing an optimized set of different text sentences for the VL based classifier to classify unclassified images.

[0013] Another embodiment of the method, wherein the adjusting includes generating an image embedding from an image encoding of a pre-classified image using a pre-trained image encoder, and generating a text embedding from a text encoding of a set of different text sentences using a pre-trained text encoder. The adjusting also includes classifying the pre-classified labeled image into one or more of a plurality of classifications based on a correlation between the image embedding and the text embedding by a VL-based classifier. The adjusting also includes determining a classification loss for the pre-classified image based at least in part on the set of text sentences. Determining the classification loss includes calculating a set of cross entropy losses for different text sentences in a superset of text sentences.

[0014] Another embodiment of the method, wherein the determining comprises iteratively repeating the adjustment of the VL-based classifier. Each iteration comprises supplying another set of different text sentences in a superset of text sentences, and each set of different text sentences comprises at least one iteration-unique text sentence in the superset. The determining further comprises: for each iteration, tracking the classification loss associated with the set of different text sentences for each iteration. The determining further comprises identifying an optimized set of different text sentences in the superset associated with a minimum classification loss of the VL-based classifier when classifying the pre-classified image.

[0015] Another embodiment of the method, wherein the minimum classification loss is associated with an optimized set of different text sentences, wherein the optimized set of different text sentences is characterized as having a minimum classification loss produced by the VL-based classifier when classifying the pre-classified image compared to other sets of the subset.

[0016] Another embodiment of the method, wherein the pre-classified images encoded by image embedding when adapting the VL-based classifier consist of twenty images or less.

[0017] According to yet another embodiment, a non-transitory machine-readable storage medium is encoded with instructions executable by one or more processors, which when executed direct the one or more processors to perform operations that facilitate optimization of a visual-language (VL) based classifier using text embedding. The operations include using a pre-trained image encoder of a visual-language model (VLM) for image embedding of pre-classified images and using a pre-trained text encoder of the VLM for text embedding of a set of different text sentences to adjust the VL based classifier. The operations also include determining an optimized set of different text sentences in a superset of text sentences. The optimized set of different text sentences has the minimum classification loss of the VL based classifier when classifying the pre-classified images. The determination includes using an automatic text sentence generator. The operations include providing an optimized set of different text sentences for the VL based classifier to classify unclassified images.

[0018] Another non-transitory machine-readable storage medium embodiment, wherein the operations further include generating an image embedding from an image encoding of a pre-classified image using a pre-trained image encoder, and generating a text embedding from a text encoding of a set of different text sentences using a pre-trained text encoder. The operations further include classifying the pre-classified labeled image into one or more of a plurality of classifications based on a correlation between the image embedding and the text embedding by a VL-based classifier. The operations further include determining a classification loss for the pre-classified image based at least in part on the set of text sentences. Determining the classification loss includes calculating a set of cross entropy losses for different text sentences in a superset of text sentences.

[0019] Another non-transitory machine-readable storage medium embodiment, wherein the operation further comprises iteratively repeating the adjustment of the VL-based classifier, wherein each iteration comprises supplying another set of different text sentences in a superset of text sentences. Each set of different text sentences comprises at least one iteration-unique text sentence in the superset. The operation further comprises: for each iteration, tracking the classification loss associated with the set of different text sentences for each iteration. The operation comprises identifying an optimized set of different text sentences in the superset associated with a minimum classification loss of the VL-based classifier when classifying a pre-classified image.

[0020] Another non-transitory machine-readable storage medium embodiment, wherein the automatic text sentence generator is selected from the group consisting of: a language model (LM), a large language model (LLM), a Markov model (MM), a recursive neural network (RNN), a transformer, a template-based generator, a rule-based generator, a reinforcement learning (RL)-based generator, and a random word generator.

[0021] Another non-transitory machine-readable storage medium embodiment, wherein the minimum classification loss is associated with an optimized set of different text sentences, wherein the optimized set of different text sentences is characterized as having a minimum classification loss produced by a VL-based classifier when classifying a pre-classified image compared to other sets of the subset.

[0022] A first aspect of the present disclosure provides a method for facilitating optimization of a vision-language (VL) based classifier using text embedding, the method comprising:

[0023] A pre-trained image encoder using a vision-language model (VLM) for image embedding of pre-classified images and a pre-trained text encoder using a VLM for text embedding of a collection of different text sentences are used to tune a VL-based classifier;

[0024] determining an optimized set of different text sentences in the superset of text sentences, wherein the optimized set of different text sentences has a minimum classification loss of the VL-based classifier when classifying the pre-classified image;

[0025] The optimized set of different text sentences is provided for the VL-based classifier to classify the unclassified image.

[0026] According to the method of the first aspect of the present disclosure, the adjusting comprises:

[0027] generating an image embedding from an image encoding of the pre-classified image using the pre-trained image encoder;

[0028] generating a text embedding from a text encoding of the set of different text sentences using the pre-trained text encoder;

[0029] classifying, by the VL-based classifier, the pre-classified image into one or more of a plurality of classifications based on correlations between the image embedding and the text embedding; and

[0030] A classification loss for the pre-classified image is determined based at least in part on the set of text sentences.

[0031] According to the method of the first aspect of the present disclosure, the adjusting comprises:

[0032] Performing image encoding on the pre-classified image using the pre-trained image encoder;

[0033] generating an image embedding vector (x) from an output of the image encoding;

[0034] Performing text encoding on the set of text sentences using the pre-trained text encoder;

[0035] Generating a text embedding vector matrix (W) from the output of the text encoding;

[0036] Correlating the image embedding vector and the text embedding vector matrix to generate a projection matrix (Wx); and

[0037] classifying, by the VL-based classifier, the pre-classified image into one or more of a plurality of classifications based on the projection matrix; and

[0038] A classification loss for the pre-classified image is determined based at least in part on the set of text sentences.

[0039] According to the method of the first aspect of the present disclosure, the determining comprises:

[0040] iteratively repeating the adjusting of the VL-based classifier, wherein each iteration comprises supplying another set of different text sentences from the superset of text sentences, wherein each set of different text sentences comprises at least one iteration-unique text sentence from the superset;

[0041] For each iteration, tracking the classification loss associated with the set of different text sentences for each iteration; and

[0042] An optimized set of different text sentences in the superset is identified, the optimized set of different text sentences associated with a minimum classification loss for the VL-based classifier when classifying the pre-classified image.

[0043] According to the method of the first aspect of the present disclosure, the determining comprises using an automatic text sentence generator.

[0044] According to the method of the first aspect of the present disclosure, the automatic text sentence generator is selected from the group consisting of: a language model (LM), a large language model (LLM), a Markov model (MM), a recursive neural network (RNN), a transformer, a template-based generator, a rule-based generator, a reinforcement learning (RL)-based generator, and a random word generator.

[0045] According to the method of the first aspect of the present disclosure, determining the classification loss includes calculating a set of cross entropy losses of different text sentences in a superset of text sentences.

[0046] A method according to the first aspect of the present disclosure, wherein the minimum classification loss is associated with an optimized set of different text sentences, and the optimized set of different text sentences is characterized as having a minimum classification loss produced by the VL-based classifier when classifying a pre-classified image compared to other sets of subsets.

[0047] The method according to the first aspect of the present disclosure further includes determining the optimal deviation ("θ") of the VL-based classifier using the optimized set of different text sentences, wherein the optimal deviation is associated with the minimum classification loss of the VL-based classifier when classifying the pre-classified image.

[0048] A second aspect of the present disclosure provides a non-transitory machine-readable storage medium encoded with instructions executable by one or more processors, which, when executed, direct the one or more processors to perform operations of the VL-based classifier generated by the method described in the first aspect of the present disclosure, wherein the VL-based classifier uses an optimized set of different text sentences for the VL-based classifier to classify unclassified images.

[0049] A third aspect of the present disclosure is a method for facilitating optimization of a vision-language (VL) based classifier using text embedding, the method comprising:

[0050] A pre-trained image encoder using a vision-language model (VLM) for image embedding of pre-classified images and a pre-trained text encoder using a VLM for text embedding of a collection of different text sentences are used to tune a VL-based classifier;

[0051] determining an optimized set of different text sentences in the superset of text sentences, wherein the optimized set of different text sentences has a minimum classification loss of the VL-based classifier when classifying the pre-classified image, wherein the determining comprises employing an automatic text sentence generator; and

[0052] The optimized set of different text sentences is provided for use by the VL-based classifier to classify the unclassified image.

[0053] The method according to the third aspect of the present disclosure, wherein the adjusting comprises:

[0054] generating an image embedding from an image encoding of the pre-classified image using the pre-trained image encoder;

[0055] generating a text embedding from a text encoding of the set of different text sentences using the pre-trained text encoder;

[0056] classifying, by the VL-based classifier, the pre-classified image into one or more of a plurality of classifications based on correlations between the image embedding and the text embedding; and

[0057] A classification loss for the pre-classified image is determined based at least in part on the set of text sentences, wherein determining the classification loss includes computing a set of cross entropy losses for different text sentences in a superset of text sentences.

[0058] The method according to the third aspect of the present disclosure, wherein the determining comprises:

[0059] iteratively repeating the adjusting of the VL-based classifier, wherein each iteration comprises supplying another set of different text sentences from the superset of text sentences, wherein each set of different text sentences comprises at least one iteration-unique text sentence from the superset;

[0060] For each iteration, tracking the classification loss associated with the set of different text sentences for each iteration; and

[0061] An optimized set of different text sentences in the superset associated with a minimum classification loss of the VL-based classifier when classifying the pre-classified image is identified.

[0062] According to the method of the third aspect of the present disclosure, the automatic text sentence generator is selected from the group consisting of: a language model (LM), a large language model (LLM), a Markov model (MM), a recursive neural network (RNN), a transformer, a template-based generator, a rule-based generator, a reinforcement learning (RL)-based generator and a random word generator.

[0063] The method according to the third aspect of the present disclosure, wherein the pre-classified images encoded by image embedding when adjusting the VL-based classifier consist of twenty or less images.

[0064] A fourth aspect of the present disclosure is a non-transitory machine-readable storage medium encoded with instructions executable by one or more processors, wherein when the instructions are executed, the one or more processors are directed to perform operations that facilitate optimization of a vision-language (VL) based classifier using text embedding, the operations comprising:

[0065] Adapting the VL-based classifier using a pre-trained image encoder of a visual-language model (VLM) for image embedding of pre-classified images and a pre-trained text encoder of the VLM for text embedding of a collection of different text sentences;

[0066] determining an optimized set of different text sentences in the superset of text sentences, wherein the optimized set of different text sentences has a minimum classification loss of the VL-based classifier when classifying the pre-classified image, wherein the determining comprises employing an automatic text sentence generator; and

[0067] The optimized set of different text sentences is provided for use by the VL-based classifier to classify the unclassified image.

[0068] According to the non-transitory machine-readable storage medium according to the fourth aspect of the present disclosure, the operations further include:

[0069] generating an image embedding from an image encoding of the pre-classified image using the pre-trained image encoder;

[0070] generating a text embedding from a text encoding of the set of different text sentences using the pre-trained text encoder;

[0071] classifying, by the VL-based classifier, the pre-classified image into one or more of a plurality of classifications based on correlations between the image embedding and the text embedding; and

[0072] A classification loss for the pre-classified image is determined based at least in part on the set of text sentences, wherein determining the classification loss includes computing a set of cross entropy losses for different text sentences in a superset of text sentences.

[0073] According to the non-transitory machine-readable storage medium according to the fourth aspect of the present disclosure, the operations further include:

[0074] iteratively repeating the adjusting of the VL-based classifier, wherein each iteration comprises supplying another set of different text sentences from the superset of text sentences, wherein each set of different text sentences comprises at least one iteration-unique text sentence from the superset;

[0075] For each iteration, tracking the classification loss associated with the set of different text sentences for each iteration; and

[0076] An optimized set of the different text sentences in the superset that are associated with a minimum classification loss of the VL-based classifier when classifying the pre-classified image is identified.

[0077] According to the non-transitory machine-readable storage medium described in the fourth aspect of the present disclosure, the automatic text sentence generator is selected from the group consisting of: a language model (LM), a large language model (LLM), a Markov model (MM), a recursive neural network (RNN), a transformer, a template-based generator, a rule-based generator, a reinforcement learning (RL)-based generator, and a random word generator.

[0078] According to the non-transitory machine-readable storage medium described in the fourth aspect of the present disclosure, the minimum classification loss is associated with an optimized set of different text sentences, and the optimized set of different text sentences is characterized as having the minimum classification loss produced by the VL-based classifier when classifying the pre-classified image compared to other sets of subsets. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] Figure 1 Illustrated is a data flow diagram of an example system that facilitates optimization of a vision-language (VL) based classifier using text embeddings according to one or more implementations described herein.

[0080] Figure 2 An example of a computer architecture for a computing system capable of performing the techniques described herein is illustrated.

[0081] Figure 3 is a flow chart illustrating a process of performing an example method that facilitates optimization of a VL-based classifier using text embeddings according to one or more implementations described herein. DETAILED DESCRIPTION

[0082] The technology described herein facilitates the optimization of visual-language (VL) based classifiers using text embeddings. Using the technology described herein, VL based classifiers can classify complex scenes, where understanding the content and its relationships is part of the classification. And this can be done so efficiently using only a small number (e.g., less than twenty) of training samples.

[0083] Referring now to the drawings, wherein like reference numerals refer to like parts throughout the several views of various systems and methods shown and described herein, the disclosed methods may be suitable for use with motor vehicles (including manual, autonomous, and semi-autonomous driving).

[0084] Figure 1 A data flow diagram of an example system 100 is illustrated that facilitates optimization of a VL-based classifier using text embeddings. According to at least some aspects of the techniques described herein, the system 100 generates a VL-based classifier that uses optimized text embeddings to classify images (or other visual information) based on complex classifications of the content of the images.

[0085] Complex classification, also known as multi-class or multi-label classification, is a machine learning (ML) task in which an input instance is assigned one or more labels from a set of multiple possible labels. Multi-class classification assigns a single label to an input instance, while multi-label classification allows for multiple labels simultaneously. Classification based on a higher-level understanding of the context of an image and the relationship between image content is an example of complex classification.

[0086] The example of complex classification described in this article involves classifying an image of a pedestrian near a road as either a pedestrian who intends to cross the road or a pedestrian who does not intend to cross the road. Therefore, the key question is whether the pedestrian in the image intends to cross the road? Examples of visual information in the image that may be relevant to answering this key question include: the pedestrian, their proximity to the road, how the pedestrian is facing the road, the pedestrian's posture, the pedestrian's gaze, other nearby actors, and the location of traffic objects (e.g., intersections, traffic lights, crosswalks).

[0087] The technique described in this article utilizes a visual-language model (VLM), also known as a vision and language model. A VLM is a computerized model that integrates visual and textual information to understand and generate content. It combines computer vision techniques with natural language processing (NLP) to bridge the gap between visual and textual data modalities.

[0088] VLMs are designed to process and generate text that describes or refers to visual content (such as images, videos, or scenes). They can perform tasks such as image captioning, visual question answering, image generation from text descriptions, and image-text matching.

[0089] These models typically have two main components: a visual encoder and a language encoder / decoder. Here, the image encoder 112 is a visual encoder and the text encoders 124, 126 are language encoders. The visual encoder processes the visual input, extracting meaningful features or representations from the visual data. This can be done using a convolutional neural network (CNN) or other computer vision models.

[0090] In some embodiments, the language decoder takes visual features and generates a text description or response. It uses natural language processing techniques and can be based on recurrent neural networks (RNNs), transformers, or other language models.

[0091] VLMs are trained on large-scale datasets containing paired visual and textual data. For example, the Image Captioning dataset consists of images paired with their corresponding text descriptions. These models are trained to learn the relationship between visual and textual data, enabling them to generate relevant and meaningful text based on visual input.

[0092] like Figure 1 As depicted in FIG. 1 , system 100 includes an image dataset 110 (such as pre-classified images y; each image is y n); a pre-trained image encoder 112 of a VLM (VLM); an image embedder 114; a pair of automatic sentence generators 120, 122 of the VLM and their accompanying pre-trained text encoders 124, 126; a text embedder 128; an image-text correlator 140; a VL-based classifier 144; and a loss checker 150. These components of the system 100 work together to optimize the VL-based classifier 144 to find an optimized set of different text sentences for the VL-based classifier to use for complex classification of unclassified images.

[0093] As part of the adaptation of the VL-based classifier 144, the system 100 employs a pre-trained image encoder 112 for image embedding of a pre-classified image (y) (via image embedder 114) and employs one or more pre-trained text encoders of a VLM (such as encoders 124, 126) for text embedding of a set of different text sentences generated by sentence generators 120, 122 (via text embedder 128).

[0094] As used herein, the training of the VL-based classifier 144 includes multiple iterations of the adjustment of the VL-based classifier 144. Each iteration has at least a set of different text sentences that are different from other text sentences in a superset of text sentences. The superset of text sentences includes a set of all different sentences that are iteratively adjusted during the overall training of the VL-based classifier 144. The VL-based classifier 144 is trained to obtain a high similarity between a positive input image (e.g., an image data set) and an optimized set of different text sentences. Here, the positive input image matches a specific classification (e.g., an intention to cross a road).

[0095] When training the VL-based classifier 144, the system 100 iteratively adjusts the classifier using an iteratively changing set of different text sentences generated by the sentence generators 120, 122. The set of different text sentences is collectively referred to as a superset of text sentences. During training, the system 100 determines an optimized set of different text sentences in the superset of text sentences. The optimized set of different text sentences is those text sentences that have the minimum classification loss of the VL-based classifier when classifying pre-classified images. That is, the optimized set of different text sentences has the lowest classification loss among the other sets in the superset.

[0096] During the training, the pre-trained image encoder 112 of the VLM receives the pre-classified (eg, labeled) images (y n). For example, this is done one image at a time. The set of pre-classified labeled images (y) can be called the ground truth. In the context of classification tasks, the ground truth labeling of an image refers to the true or correct class label associated with a data point in the labeled dataset. It represents the known or annotated class of the data point.

[0097] Pre-trained (also called "frozen") encoders are part of the VLM. Pre-trained encoders are trained on large datasets in an unsupervised manner to learn meaningful visual and data representations. These representations capture contextual information and can be used for transfer learning on various natural language understanding tasks.

[0098] During training and evaluation of a classifier, the pre-classified labels are used as reference or target values ​​to which the classifier's predictions are compared. A classifier is designed to learn from the training data and accurately predict the class labels of new, unseen data points.

[0099] For example, suppose you have a dataset with labeled images, with labels indicating whether each image depicts a pedestrian (C) or (N) intending to cross the road. Pre-classified labels will assign the correct category, such as "C" or "N", to each image. These labels are typically provided by someone who has manually annotated the dataset or using some other reliable method.

[0100] During training, the classifier attempts to learn the underlying patterns and features in the data that correlate to the correct class label. The classifier adjusts its internal parameters based on the difference between its predictions and the pre-classified labels, aiming to minimize the prediction error or loss.

[0101] The image embedder 114 receives the output from the image encoder 112 and as part of the conditioning, it generates an image embedding vector (x). An embedding vector (i.e., embedding) is a digital representation of an object or entity in an ML model. It is typically used to represent discrete or categorical variables, such as words or images, in a continuous and dense vector space.

[0102] The purpose of embedding vectors is to capture the underlying relationships and semantics of the objects represented. By mapping objects into a continuous vector space, embedding enables ML models to make inferences and predictions based on the relationships between vectors.

[0103] Text embeddings are one of the most common types of embeddings. They are used to represent words in natural language processing tasks. Word embeddings encode semantic and syntactic information about words by considering their co-occurrence patterns in a large corpus of text.

[0104] Embedding vectors can also represent other types of objects, such as images. In image processing, convolutional neural networks (CNNs) can be used to learn embeddings that capture the visual features of an image. These embeddings can be used for tasks such as image classification, object detection, or image retrieval.

[0105] As part of the adjustment, each pre-trained text encoder 124, 126 of the VLM receives a text sentence (e.g., s1 and s2) generated by its associated automatic sentence generator 120, 122, respectively. As used herein, a text sentence is a collection of multiple words (e.g., English) or other semantic indicators / symbols. Text sentences do not necessarily need to comply with language syntax or grammatical rules. Examples of acceptable text sentences include: "a photo of a person running with his back to the camera", "a person on the sidewalk staring directly at a vehicle", "talking on the phone while pushing a stroller", and "walking with a bicycle helmet taken off".

[0106] The automatic sentence generators 120, 122 can be implemented in various ways. Based on their underlying technology and capabilities, they can be classified into several types. For example, the automatic sentence generator can be implemented in one or more of the following ways: language model (LM), large language model (LLM), Markov model (MM), recursive neural network (RNN), transformer, template-based generator, rule-based generator, reinforcement learning (RL)-based generator, or random word generator.

[0107] Automatic sentence generators of LM and LLM generate text by predicting the next word or sequence of words based on the input context. They can be trained on large amounts of text data and are commonly used for tasks such as language modeling, text completion, and text generation.

[0108] MM's automatic sentence generator generates text based on the statistical properties of the input data. They use Markov chains to model the likelihood of transitioning from one word or state to another. Markov models are relatively simple, but can generate coherent text based on patterns learned from training data.

[0109] Automatic sentence generators with RNNs generate text by processing sequential data. Their memory components allow them to capture dependencies across time steps. RNNs are often used for text generation, machine translation, and sentiment analysis tasks.

[0110] The Transformer's automatic sentence generator is good at capturing long-range dependencies and has been successfully applied to tasks such as machine translation, question answering, and text generation.

[0111] Template-based generators use predefined templates or patterns to generate text. They can fill in placeholders or variables within the template with appropriate words or phrases based on certain rules or conditions. Template-based generators are often used to generate structured text, such as form letters or repetitive content.

[0112] Rule-based generators generate text based on explicit rules or instructions. These rules can specify the structure, syntax, or logic that controls the generation process. Rule-based generators are often used for specific domains or tasks where the generation process can be clearly defined.

[0113] Reinforcement learning (RL) based generators use reinforcement learning techniques to generate text. The model learns from reward signals from evaluators based on the quality or relevance of the generated text. RL based generators can be trained to optimize objectives such as fluency, coherence, or adherence to specific guidelines.

[0114] Random word generators do exactly that. They can pick up words at random from one or more sets of words. These words are usually carefully crafted to be related to the intended classification.

[0115] In the training of the VL-based classifier 144, the automatic sentence generators iteratively adjust or change the sentences they generate for each iteration. The loop 130 of the automatic sentence generators 120, 122 represents this iterative sentence adjustment.

[0116] In some embodiments, the two automatic sentence generators have the same prompt sentence. For example, the prompt sentence may be "photo of a person". In other cases, they may be given different prompt sentences.

[0117] In some implementations, the system 100 adjusts one sentence at a time by alternating between adding one or more words in the sentence or changing one or more words in the sentence. For example, in one iteration, the automatic sentence generator 120 adds words to sentence s1, while the automatic sentence generator 122 leaves sentence s2 unchanged. In the next iteration of training, the automatic sentence generator 122 adds words to sentence s2, while the automatic sentence generator 120 leaves sentence s1 unchanged.

[0118] As described, system 100 includes two different sentences (e.g., s1 and s2) for its source of text embeddings. This described embodiment has a binary classification. Each image depicts a pedestrian intending to cross the road (C) or not intending to cross the road (N). However, other embodiments of the technology described herein may employ more than just two classifications. Therefore, a collection of different sentences can include a matching number of sentences as its source of text embeddings. Therefore, the number of sentences in the collection matches the number of dimensions of the feature space on which the classifier acts.

[0119] The text encoders 124, 126 receive text sentences (e.g., s1 and s2) from their associated automatic sentence generators 120, 122, respectively. Each encoder encodes its associated output, which is mathematically indicated by G. For example, the text encoder 124 generates an output w1=G(s1) based on the sentence s1 from the sentence generator 120, and the text encoder 126 generates an output w2=G(s2) based on the sentence s2 from the sentence generator 122.

[0120] The text embedder 128 receives the outputs from the text encoders 124, 126, and as part of the adjustment, it generates a text embedding vector matrix (W) from the output of each text encoder. Therefore, the text embedding vector matrix can be represented as a matrix of W(s1, s1) = G(s1) and G(s2).

[0121] As part of the adjustment, the image-text correlator 140 correlates the image embedding vector and the text embedding vector matrix to produce a projection matrix (Wx). The projection matrix is ​​a square matrix that transforms a vector to a lower dimensional subspace by projecting the vector to a smaller space.

[0122] The projection matrix (Wx) is a two-dimensional space with two axes: the X-axis is G(s1)x, and the Y-axis is G(s2)x. This two-dimensional space is a high-level semantic feature space intuitively depicted in the high-level semantic feature space diagram 142. In general, a high-level semantic feature space refers to a data representation that captures the meaning or semantic content of data at a higher level of abstraction. In various fields including natural language processing (NLP), computer vision, and audio processing, a high-level semantic feature space extracts and represents basic semantic information from raw data.

[0123] High-level semantic feature spaces in natural language processing can involve word embeddings or contextual word representations obtained from pre-trained language models. These embeddings encode the meaning of words or sentences in dense vector spaces, enabling better semantic understanding and downstream NLP tasks such as text classification, sentiment analysis, and question answering.

[0124] In computer vision, high-level semantic feature spaces can be obtained by extracting features from deep CNNs. These CNNs are trained on large image datasets and learn to recognize high-level visual patterns, objects, and concepts. Using these features, image data can be represented in a compact and semantically meaningful space, enabling tasks such as image classification, object detection, and image captioning.

[0125] The high-level semantic feature space chart 142 shows that the X-axis is G(s1)x and the Y-axis is G(s2)x. The letter C ("C") in the chart represents the position in the feature space of the image in the pre-classified ground truth image (y) that represents a pedestrian who does intend to cross the road. The letter N ("N") in the chart represents the position in the feature space of the image in the pre-classified ground truth image that represents a pedestrian who does not intend to cross the road. Of course, this is known because the input image is the ground truth and is therefore pre-classified.

[0126] As part of the adjustment, the VL-based classifier 144 classifies the pre-classified labeled images from the image dataset 110 into one or more of a plurality of classifications based on the projection matrix. This can be accomplished using a multi-layer perceptron (MLP) with parameters "theta" (θ) and represented by f(Wx,θ). An MLP is an artificial neural network with multiple layers of interconnected neurons. It consists of an input layer that receives data, one or more hidden layers that process the data using activation functions, and an output layer that generates predictions. MLPs learn from data by adjusting weights and biases via optimization methods such as gradient descent and back-propagation. They are versatile models for classification, regression, and pattern recognition tasks, but other deep learning architectures such as CNNs and RNNs are also becoming increasingly popular in specific applications.

[0127] The result of this classification is seen in decision boundary graph 146, which is the same as graph 142 except that decision boundary 148 is shown. The decision boundary is a graphical representation of the classification f(Wx, θ). Graphically, decision boundary 148 separates Cs from Ns.

[0128] The VL-based classifier 144 may also be referred to as a classification network. Such a classifier is an ML model designed to solve classification problems. Classification is the task of assigning input data points to predefined categories or classes.

[0129] The architecture of a classifier usually consists of an input layer, one or more hidden layers, and an output layer. Each layer consists of multiple interconnected nodes (called neurons). The input layer receives the raw input data, which is then processed by the hidden layer, and finally the output layer produces the classification result.

[0130] The hidden layers of the classifier perform a series of calculations that transform the input data in a nonlinear way to extract relevant features and patterns. Each neuron in the hidden layer takes a weighted sum of the outputs from the previous layer, applies an activation function to introduce nonlinearity, and passes the result to the next layer. This process allows the network to learn complex relationships between input data and corresponding class labels.

[0131] The output layer of a classifier usually consists of one neuron per class, using an activation function that produces a probability score for each class. The class with the highest probability is the predicted class for a given input.

[0132] During training, a classifier (such as the VL-based classifier 144) learns the optimal weights and biases of its neurons by iteratively adjusting the weights and biases of the neurons to minimize a loss function that quantifies the difference between the predicted class labels and the true class labels in the training data. The classifier learns from a training data set (such as a ground truth image y) in which each input data point is associated with a known class label. During training, the classifier adjusts its internal parameters based on the input-output pairs in the training data to improve its ability to correctly classify unseen data.

[0133] Once trained, a classification neural network can predict the class label of new, unseen data points by feeding them into the network and obtaining the output from the output layer.

[0134] In the context of a classifier, "theta" (θ) generally refers to the parameters or weights of a classification network (i.e., a classifier). Here, "theta" ("θ") is described as the optimal deviation associated with the minimum classification loss of a VL-based classifier when classifying a pre-classified image.

[0135] These parameters are learned during the training process. In a feedforward neural network (a common type of classification network), the parameters include weights and biases associated with the connections between neurons. Each connection has a weight value that determines the strength of the connection, and each neuron has a bias term that allows the network to learn different representations.

[0136] The weights and biases together form a parameter vector, often denoted as "theta" (θ). This vector represents the adjustable parameters in a classification network (i.e., a classifier). The value of the optimal bias (θ) is optimized during training to minimize a loss function that quantifies the difference between the predicted output and the true label.

[0137] By updating the value of the optimal bias (θ), the classification network (i.e., classifier) ​​learns to adjust the strength and bias of the connections of the neurons, thereby gradually improving its ability to accurately classify the input data. The technique described in this article utilizes an optimized set of different text sentences to determine the optimal bias ("θ") for the VL-based classifier. The optimal bias is associated with the minimum classification loss of the VL-based classifier when classifying pre-classified images.

[0138] In the adjustment, the loss checker 150 is based at least in part on different sentences (e.g., s lThe loss checker 150 uses a cross entropy (CE) loss calculation to determine the classification loss or error. Cross entropy loss, also known as logarithmic loss or logical loss, is a commonly used loss function in classification tasks, particularly in binary or multi-class classification problems. It quantifies the difference between the predicted probability and the true class label.

[0139] In binary classification, the cross entropy loss measures the difference between the predicted probability of the positive class and the true binary label, which can take values ​​of 0 or 1.

[0140] In multi-class classification where there are more than two classes, the cross entropy loss is extended to handle multiple classes. Each class has its own predicted probability, and the true label is represented as a one-hot encoded vector.

[0141] The cross entropy loss function encourages the predicted probabilities to be as close to the true labels as possible. It penalizes confident wrong predictions (predicted probabilities far from the true labels) and uncertain predictions (predicted probabilities close to 0.5).

[0142] During the training process, the cross entropy loss is used as an optimization target to guide the learning of the classifier. The goal is to minimize the average cross entropy loss across training samples. For example, to minimize the cross entropy loss between the output of the classification probability (e.g., from the VL-based classifier 144) and the pre-classified label probability.

[0143] During training, the loss checker 150 employs backpropagation, which is a supervised learning method used to train artificial neural networks. It involves a forward pass to calculate predictions, followed by a backward pass to calculate the gradients of the weights and biases with respect to the loss function. These gradients are used to update the parameters in the network with the goal of minimizing the error between the predicted output and the actual output. By iteratively adjusting the weights and biases, backpropagation helps the neural network learn from the data and make accurate predictions for new examples.

[0144] like Figure 1 As shown in , equation 160 depicts the optimization calculation performed for each iteration of classifier adjustment. This formula helps calculate the sentence combination (e.g., s1 and s2) that results in the lowest (i.e., minimum) classification loss among all attempted sentence combinations. Using this formula helps find the optimized set of different sentences in the superset (all combinations used in training) based on the optimized set that produces the lowest (i.e., minimum) classification loss in the superset. The cross entropy (CE) loss calculation determines the classification loss.

[0145] Figure 2An example of a computer architecture of a computing system 200 capable of performing the techniques described herein is illustrated. The computer architecture in the figure illustrates a typical computer, server computer, workstation, desktop computer, laptop computer, tablet computer, network appliance, e-reader, smart phone, embedded system, or other computing device. Although shown as a single discrete device, computing system 200 may be part of a distributed and interconnected collection of components that perform the same function.

[0146] The computing system 200 includes a processor 202 (e.g., a central processing unit or "CPU"), system storage (e.g., memory) 204, input / output (I / O) devices 206 (such as a display, keyboard, mouse, microphone, camera and associated controllers), a secondary storage system 208 (e.g., a hard drive), and various other subsystems 210. In various embodiments, the computing system 200 also includes a communication ("comm") port 212 that is operable to connect to a communication ("comm") system 220. The computing system 200 may include or be connected to a database 222 containing an image data set. The foregoing components may be interconnected via one or more buses 216 and / or communication systems 220.

[0147] The communication system 220 implements internal communication within the computing system 200 and external wireless communication with devices and networks outside the system (such as with the external communication network 230). Internal communication can be, for example, via a local area network. External wireless communication can include, for example, one or more of the following: satellite communication, WI-FI TM 、BLUETOOTH TM , cellular communications, radio communications and / or Internet communications.

[0148] System memory 204 may store data and machine-readable instructions (e.g., computer-readable instructions). The machine-readable instructions may configure computing system 200. The machine-readable instructions may include one or more instruction modules. The instruction modules may include computer program modules. The instruction modules may include one or more of VLM 230, trainer 232, VL-based classifier 234, sentence generation manager 236, optimizer 238, and / or other instruction-based modules.

[0149] Although the modules in the example depicted in the figure are implemented using machine-readable instructions, other similar functional modules can be implemented using few or no machine-readable instructions. Such modules are implemented via hardware (e.g., circuits) and analog and / or digital signals. In other cases, such modules can be implemented via a combination of hardware and machine-implemented instructions.

[0150] VLM 230 is a visual-language model (VLM). VLM 230 is configured to include the functionality of a pre-trained image encoder (such as image encoder 112) and one or more pre-trained text encoders (such as text encoders 122, 124).

[0151] The trainer 232 is configured to manage the training of the VL-based classifier 234. The trainer 232 is configured to include the functionality of an image embedder (such as the image embedder 114) and a text embedder (such as the text embedder 128). The trainer 232 is also configured to include the functionality of an image-text correlator (such as the image-text correlator 140).

[0152] The VL-based classifier 234 is configured to classify an input image (such as a pre-classified labeled image from the image dataset 110) into one or more of a plurality of categories based on a projection matrix (Wx). The projection matrix is ​​a correlation of the image embedding vector (x) and the text embedding vector matrix (W) from the output of the text encoder for different sentences (s1 and s2).

[0153] Once training is completed, the VL-based classifier 234 is optimized using parameters 240 stored in a local memory, a database, and / or an external device on the external communication network 230. The stored parameters 240 include the optimized sentences (e.g., s1 and s2) and other classifier parameters (e.g., θ). Using these stored parameters 240, the trained VL-based classifier 234 can effectively classify the input image, thereby confirming that there is a high degree of similarity between the input image and the optimized sentence. Therefore, the trained VL-based classifier 144 can successfully perform complex classification of unclassified images.

[0154] As part of the training of the VL-based classifier 234, the sentence generation manager 236 manages and guides the automatic sentence generator to iteratively adjust or change the input sentences. In some embodiments, the sentence generation manager 236 adjusts one sentence at a time by adding one or more words in the sentence or changing one or more words in the sentence alternately. For example, in one iteration, the sentence generation manager 236 guides the automatic sentence generator to add words to the first input sentence (e.g., s1) while keeping the second input sentence (e.g., s2) unchanged. In the next iteration of training, the sentence generation manager 236 guides the automatic sentence generator to add words to the second input sentence (e.g., s2) while keeping the first input sentence (e.g., s1) unchanged.

[0155] In some implementations, the one or more automatic sentence generators are modules of data and machine-readable instructions stored in the system memory 204. In other implementations, the one or more automatic sentence generators are external modules or services accessed via the external communication network 230.

[0156] As part of the training of the VL-based classifier 234, the optimizer 238 determines an optimized set of different text sentences in the superset of text sentences. The optimized set of different text sentences are those text sentences that have the lowest classification loss of the VL-based classifier when classifying the pre-classified image. That is, the optimized set of different text sentences has the lowest classification loss among the other sets in the superset.

[0157] Figure 3 is a flow chart illustrating a process 300 of performing an example method that facilitates optimization of a VL-based classifier utilizing text embeddings. For ease of illustration, process 300 may be described as being performed by a system described herein. Such a system may be, for example, computing system 200. Process 300 represents training a VL-based classifier to find optimized parameters, such as text embeddings.

[0158] At operation 310, the system uses the VLM to perform feature extraction for each training iteration. This operation includes using a pre-trained image encoder to generate image embeddings based on image encodings of pre-classified images from the image dataset 312. As depicted, y n represents the sample image used for each iteration of classifier training. n It is a sample image among multiple (ie, n) images in the pre-classified images.

[0159] Operation 310 also includes using a pre-trained text encoder to extract the text from different text sentences (e.g., s l and s2) generate text embeddings. As described, the automatic sentence generator 314 generates iteratively adjusted or changed sentences for each iteration. In some embodiments, the automatic sentence generator 314 is provided with the same prompt sentence. In some embodiments, the automatic sentence generator 314 alternately adjusts one sentence at a time by adding one or more words in the sentence or changing one or more words in the sentence.

[0160] At operation 320, the system performs VL-based classification for each iteration. This includes classifying the pre-classified labeled images into one or more of a plurality of classifications based on, for example, the correlation of image embeddings and text embeddings.

[0161] In general, each iteration of operations 310 and 320 can be described as the system adjusting the VL-based classifier by employing the VLM's pre-trained image encoder to perform image embedding on pre-classified images and employing the VLM's pre-trained text encoder to perform text embedding on each iterative set of different text sentences.

[0162] At operation 330, the system performs optimization calculations based on the classification loss of the pre-classified images for each iteration, the classification loss being based at least in part on the classification loss of different sentences (e.g., s l Operation 330 may calculate a cross entropy (CE) loss to determine the classification loss or error.

[0163] Operation 330 is performed, for example, according to the above-mentioned Figure 1 The optimization calculation is performed in equation 160 shown in. This calculation produces the parameters 332 such as s1, s2 and θ that are used for the iterative adjustment of the classifier.

[0164] At operation 340, the system determines whether the parameters 332 are now optimized. This determination can be based on a lack of improvement in the parameters 332 in subsequent iterations. The best or optimized parameters are the parameters that minimize the classification loss, such as calculated using equation 160. Thus, the system identifies the combination of iterative parameters that produces the minimum classification loss.

[0165] If the parameters 332 have not been optimized, the process 300 returns to operation 310 to perform another iteration. If the parameters 332 have not been optimized, the system identifies the optimized parameters 342 (eg, Opt(s1, s2, θ)).

[0166] In general, operations 330 and 340 may be described as the system determining an optimized set of different text sentences in a superset of text sentences, wherein the optimized set of different text sentences has a minimum classification loss for a VL-based classifier when classifying a pre-classified image.

[0167] More specifically, operations 330 and 340 can be further described as the system iteratively repeating the adjustment of the VL-based classifier. Each iteration includes supplying another set of different text sentences in the superset of text sentences. Each set of different text sentences includes at least one iteratively unique text sentence in the superset.

[0168] In addition, for each iteration, the system tracks the classification loss associated with the set of different text sentences for each iteration. It identifies the optimal set of different text sentences in the superset that is associated with the minimum classification loss of the VL-based classifier when classifying the pre-classified images.

[0169] At operation 344, the system provides optimized parameters 342. For example, the system can provide an optimized set of different text sentences (e.g., Opt (s1, s2,)) for the trained VL-based classifier to use to classify the unclassified image. The trained VL-based classifier is depicted as box 350.

[0170] The above description is intended to be illustrative and not restrictive. When the size and type of the material described herein are intended to be illustrative, they are by no means restrictive and are exemplary embodiments. In the appended claims, the use of the terms "first", "second", "top", "bottom", etc. is only used as a mark and is not intended to impose a number or position requirement on their objects. As used herein, the element or step recorded in the singular and preceded by the word "one" or "one" should be understood as not excluding a plurality of such elements or steps, unless such exclusion is clearly stated. In addition, the phrase "at least one of A and B" and the phrase "A and / or B" should all be understood to mean "only A, only B, or both A and B". In addition, unless clearly stated to the contrary, the embodiment of "including" or "having" one or more elements with a specific attribute can include additional such elements without the attribute. And when broad descriptive adverbs such as "substantially" and "generally" are used herein to modify adjectives, these adverbs mean "most", "mainly", "most", "to a large extent", "to a large extent" and / or "at least 51% to 99% of the possible range of 100%, and do not necessarily mean "perfectly", "completely", "strictly", "entirely" or "100%". In addition, the word "close" may be used herein to describe the position of one object or a part thereof relative to another object or a part thereof, and / or to describe the positional relationship of two objects or their respective parts relative to each other, and may mean "near", "adjacent", "close to", "close to", "at", etc. And, the phrase "approximately equal to" used herein may mean one or more of "completely equal to", "almost equal to", "equal to somewhere between 90% and 110%", etc.

[0171] This written description uses examples, including the best mode, to enable those skilled in the art to make and use devices, systems, and compositions of matter according to the present disclosure, and to perform methods according to the present disclosure. The following claims, including equivalents, define the scope of the present disclosure.

Claims

1. A method for facilitating optimization of a vision-language (VL) based classifier using text embeddings, the method comprising: A pre-trained image encoder using a vision-language model (VLM) for image embedding of pre-classified images and a pre-trained text encoder using a VLM for text embedding of a collection of different text sentences are used to tune a VL-based classifier; determining an optimized set of different text sentences in the superset of text sentences, wherein the optimized set of different text sentences has a minimum classification loss of the VL-based classifier when classifying the pre-classified image; The optimized set of different text sentences is provided for the VL-based classifier to classify the unclassified image.

2. The method of claim 1, wherein the adjusting comprises: generating an image embedding from an image encoding of the pre-classified image using the pre-trained image encoder; generating a text embedding from a text encoding of the set of different text sentences using the pre-trained text encoder; classifying, by the VL-based classifier, the pre-classified image into one or more of a plurality of classifications based on correlations between the image embedding and the text embedding; and A classification loss for the pre-classified image is determined based at least in part on the set of text sentences.

3. The method of claim 1, wherein the adjusting comprises: Performing image encoding on the pre-classified image using the pre-trained image encoder; generating an image embedding vector (x) from an output of the image encoding; Performing text encoding on the set of text sentences using the pre-trained text encoder; Generating a text embedding vector matrix (W) from the output of the text encoding; Correlating the image embedding vector and the text embedding vector matrix to generate a projection matrix (Wx); as well as Classifying the pre-classified image into one or more of a plurality of classifications based on the projection matrix by the VL-based classifier; as well as A classification loss for the pre-classified image is determined based at least in part on the set of text sentences.

4. The method of claim 1, wherein the determining comprises: iteratively repeating the adjusting of the VL-based classifier, wherein each iteration comprises supplying another set of different text sentences from the superset of text sentences, wherein each set of different text sentences comprises at least one iteration-unique text sentence from the superset; For each iteration, keep track of the classification loss associated with the set of different text sentences for each iteration; as well as An optimized set of different text sentences in the superset is identified, the optimized set of different text sentences associated with a minimum classification loss for the VL-based classifier when classifying the pre-classified image. The method of claim 1 , wherein said determining comprises employing an automatic text sentence generator.

6. The method of claim 5, wherein the automatic text sentence generator is selected from the group consisting of: a language model (LM), a large language model (LLM), a Markov model (MM), a recursive neural network (RNN), a transformer, a template-based generator, a rule-based generator, a reinforcement learning (RL)-based generator, and a random word generator.

7. The method of claim 2, wherein determining the classification loss comprises calculating a set of cross entropy losses for different text sentences in a superset of text sentences.

8. The method of claim 1, wherein the minimum classification loss is associated with an optimized set of the different text sentences, the optimized set of the different text sentences being characterized as having a minimum classification loss produced by the VL-based classifier when classifying the pre-classified image compared to other sets of subsets.

9. The method of claim 1 , further comprising determining an optimal bias (“θ”) for the VL-based classifier using the optimized set of different text sentences, wherein the optimal bias is associated with a minimum classification loss for the VL-based classifier when classifying a pre-classified image.

10. A non-transitory machine-readable storage medium encoded with instructions executable by one or more processors, which, when executed, direct the one or more processors to perform operations of the VL-based classifier generated by the method of claim 1, wherein the VL-based classifier uses the optimized set of different text sentences for the VL-based classifier to classify unclassified images.

Images