Open vocabulary object detection based on frozen vision and language model

Abstract

An example method is provided for training a detector head to perform object detection on a training object category based on a frozen vision and language model (VLM). The method includes receiving the frozen VLM that is pre-trained over a plurality of image-text pairs. The method includes determining, for an image embedding generated by a pre-trained image encoder of the frozen VLM and by the detector head, a detection region embedding indicative of one or more regions of interest in an image. The method includes generating, by a pre-trained text encoder of the frozen VLM, a text embedding of the training object category. The method includes predicting, by the detector head, an object from a target object vocabulary associated with the training object category based on the detection area embedding and the text embedding of the training object category. The method includes providing the pre-trained frozen VLM and a trained detector head.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS / INCORPORATED BY REFERENCE

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 367,178, filed on June 28, 2022, which is hereby incorporated by reference in its entirety. Background Art

[0003] Object detection is a visual task that is usually based on algorithms to locate and identify objects in an image. Object detection requires the identification and localization of objects across various scales. Summary of the invention

[0004] Some object detection models rely on trained vocabularies and are therefore not suitable for open vocabulary object detection. In general, open vocabulary object detection can leverage other sources of supervision, such as image captions or visual and language pre-training. Due to the need for region-level generalization, such methods typically involve knowledge distillation, region distillation on external data, or pre-training using image-level captions in addition to standard detection training. Some methods rely on pre-trained vision and language models (VLMs) for generalization. VLMs are able to generate rich knowledge and powerful representations for vision and language domains. However, in many VLMs, the entire detector head may need to be trained from scratch. Some VLMs rely on separate pre-training and fine-tuning processes. However, these models may lack scalability, and retraining, pre-training, and / or fine-tuning of detection may be computationally resource-intensive.

[0005] Therefore, there is a need for a simple and scalable open vocabulary detection method that can extract local sensitive information using a lightweight detector head. Specifically, as described in this paper, a detector head can be trained on a frozen VLM backbone, and the detection scores of the detector head can be combined with the corresponding VLM predictions at test time.

[0006] In one aspect, a computer-implemented method for training a detector head to perform object detection for a training object class based on a frozen vision and language model (VLM) is provided. The method includes receiving, by a computing device, a frozen VLM pre-trained on a plurality of image-text pairs. The method also includes determining, for an image encoder pre-trained by the frozen VLM and an image embedding generated by the detector head, a detection region embedding indicating one or more regions of interest in the image. The method additionally includes generating, by the pre-trained text encoder of the frozen VLM, a text embedding for the training object class. The method further includes predicting, by the detector head, an object from a target object vocabulary associated with the training object class based on the detection region embedding and the text embedding for the training object class. The method also includes providing, by a computing device, the pre-trained frozen VLM and the trained detector head.

[0007] In a second aspect, a computing device is provided. The computing device includes one or more processors and a data storage device. The data storage device has computer executable instructions stored thereon, which, when executed by the one or more processors, cause the computing device to perform functions of training a detector head to perform object detection on a training object category based on a frozen visual and language model (VLM). The functions include: receiving, by the computing device, a frozen VLM pre-trained on a plurality of image-text pairs; determining, for an image encoder pre-trained by the frozen VLM and an image embedding generated by the detector head, a detection region embedding indicating one or more regions of interest in the image; generating a text embedding of the training object category by the pre-trained text encoder of the frozen VLM; predicting, by the detector head, an object from a target object vocabulary associated with the training object category based on the detection region embedding and the text embedding of the training object category; and providing, by the computing device, the pre-trained frozen VLM and the trained detector head.

[0008] In a third aspect, a computer program is provided. The computer program includes instructions that, when executed by a computer, cause the computer to perform functions of training a detector head to perform object detection on a training object category based on a frozen visual and language model (VLM). The functions include: receiving, by a computing device, a frozen VLM pre-trained on a plurality of image-text pairs; determining, for an image encoder pre-trained by the frozen VLM and an image embedding generated by the detector head, a detection region embedding indicating one or more regions of interest in the image; generating a text embedding of the training object category by the pre-trained text encoder of the frozen VLM; predicting, by the detector head, an object from a target object vocabulary associated with the training object category based on the detection region embedding and the text embedding of the training object category; and providing, by a computing device, the pre-trained frozen VLM and the trained detector head.

[0009] In a fourth aspect, an article of manufacture is provided. The article of manufacture includes one or more computer-readable media having computer-readable instructions stored thereon, which, when executed by one or more processors of a computing device, cause the computing device to perform functions of training a detector head to perform object detection for a training object category based on a frozen visual and language model (VLM). The functions include: receiving, by the computing device, a frozen VLM pre-trained on a plurality of image-text pairs; determining, for an image encoder pre-trained by the frozen VLM and an image embedding generated by the detector head, a detection region embedding indicating one or more regions of interest in the image; generating a text embedding for the training object category by the pre-trained text encoder of the frozen VLM; predicting, by the detector head, an object from a target object vocabulary associated with the training object category based on the detection region embedding and the text embedding for the training object category; and providing, by the computing device, the pre-trained frozen VLM and the trained detector head.

[0010] In a fifth aspect, a system for performing the function of training a detector head based on a frozen visual and language model (VLM) to perform object detection on a training object category is provided. The system includes: a component for receiving a frozen VLM pre-trained on multiple image-text pairs by a computing device; a component for determining a detection region embedding indicating one or more regions of interest in an image for an image encoder pre-trained by the frozen VLM and an image embedding generated by the detector head; a component for generating a text embedding of a training object category by a pre-trained text encoder of the frozen VLM; a component for predicting an object from a target object vocabulary associated with the training object category by the detector head based on the detection region embedding and the text embedding of the training object category; and a component for providing the pre-trained frozen VLM and the trained detector head by a computing device.

[0011] In a sixth aspect, a computer-implemented method for applying a trained detector head to perform object detection on a training object category based on a frozen vision and language model (VLM) is provided. The method includes receiving an input image by a computing device. The method also includes applying a trained neural network for object detection, wherein the neural network includes a frozen VLM pre-trained on multiple image-text pairs, and a trained detector head associated with the pre-trained frozen VLM and pre-trained on the training object category. The method further includes: for an image encoder pre-trained by the frozen VLM and an image embedding generated by the detector head, determining a detection region embedding indicating one or more regions of interest in the input image. The method also includes: predicting, by the detector head, an object from a target object vocabulary associated with the training object category based on the detection region embedding and the text embedding of the training object category. The method further includes providing, by the computing device, an object from the target object vocabulary to the input image.

[0012] In a seventh aspect, a computing device for applying a trained detector head to perform object detection on a training object category based on a frozen visual and language model (VLM) is provided. The computing device includes one or more processors and a data storage device. The data storage device has computer executable instructions stored thereon, and the computer executable instructions, when executed by the one or more processors, cause the computing device to perform functions. The functions include: receiving an input image by the computing device; applying a trained neural network for object detection, wherein the neural network includes a frozen VLM pre-trained on multiple image-text pairs, and a trained detector head associated with the pre-trained frozen VLM and pre-trained on the training object category; determining a detection region embedding indicating one or more regions of interest in the input image for an image encoder pre-trained by the frozen VLM and an image embedding generated by the detector head; predicting, by the detector head, an object from a target object vocabulary associated with the training object category based on the detection region embedding and the text embedding of the training object category; and providing, by the computing device, an object from the target object vocabulary to the input image.

[0013] In an eighth aspect, a computer program for applying a trained detector head to perform object detection on a training object category based on a frozen visual and language model (VLM) is provided. The computer program includes instructions that, when executed by a computer, cause the computer to perform functions. The functions include: receiving an input image by a computing device; applying a trained neural network for object detection, wherein the neural network includes a frozen VLM pre-trained on a plurality of image-text pairs, and a trained detector head associated with the pre-trained frozen VLM and pre-trained on the training object category; determining a detection region embedding indicating one or more regions of interest in the input image for an image encoder pre-trained by the frozen VLM and an image embedding generated by the detector head; predicting, by the detector head, an object from a target object vocabulary associated with the training object category based on the detection region embedding and the text embedding of the training object category; and providing, by the computing device, an object from the target object vocabulary to the input image.

[0014] In a ninth aspect, there is provided an article for applying a trained detector head to perform object detection for a training object category based on a frozen vision and language model (VLM). The article comprises one or more computer-readable media having computer-readable instructions stored thereon, which, when executed by one or more processors of a computing device, cause the computing device to perform functions. The functions include: receiving an input image by the computing device; applying a trained neural network for object detection, wherein the neural network includes a frozen VLM pre-trained on a plurality of image-text pairs, and a trained detector head associated with the pre-trained frozen VLM and pre-trained on the training object category; determining, for the image encoder pre-trained by the frozen VLM and the image embedding generated by the detector head, a detection region embedding indicating one or more regions of interest in the input image; predicting, by the detector head, an object from a target object vocabulary associated with the training object category based on the detection region embedding and the text embedding of the training object category; and providing, by the computing device, an object from the target object vocabulary to the input image.

[0015] In a tenth aspect, a system for applying a trained detector head to perform object detection on a training object category based on a frozen visual and language model (VLM) is provided. The system includes: a component for receiving an input image by a computing device; applying a trained neural network for object detection, wherein the neural network includes a frozen VLM pre-trained on a plurality of image-text pairs, and a trained detector head associated with the pre-trained frozen VLM and pre-trained on the training object category; a component for determining a detection region embedding indicating one or more regions of interest in the input image for an image encoder pre-trained by the frozen VLM and an image embedding generated by the detector head; a component for predicting, by the detector head, an object from a target object vocabulary associated with the training object category based on the detection region embedding and a text embedding of the training object category; and a component for providing, by the computing device, an object from the target object vocabulary to the input image.

[0016] The foregoing summary is illustrative only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments, and features described above, further aspects, embodiments, and features will become apparent by reference to the drawings and the following detailed description and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a diagram illustrating example training stages of a neural network according to an example embodiment.

[0018] Figure 2 is a diagram illustrating example inference stages of a neural network according to an example embodiment.

[0019] Figure 3 is a table showing comparison between various object detection models according to example embodiments.

[0020] Figure 4 is another table illustrating comparisons between various object detection models according to example embodiments.

[0021] Figure 5 is a table showing example results on various test data sets according to example embodiments.

[0022] Figure 6 is a table showing example results of performance and training cost tradeoffs according to example embodiments.

[0023] Figure 7 Examples of object detection tasks on various images according to example embodiments are shown.

[0024] Fig. 8A is a table showing example results of VLM score weights for new classes, according to an example embodiment.

[0025] Figure 8B is a table showing example results of fusing VLM and detection scores using arithmetic and geometric means, according to an example embodiment.

[0026] Fig.9A is a table illustrating fractional fusion parameters according to an example embodiment.

[0027] Fig. 9B is a table illustrating fractional fusion parameters according to an example embodiment.

[0028] Fig. 9C is a table illustrating fractional fusion parameters according to an example embodiment.

[0029] Fig. 10A An example table showing feature pyramid capacities is shown according to an example embodiment.

[0030] Fig. 10B An example table illustrating the impact of context weights on open vocabulary detection is shown according to an example embodiment.

[0031] Fig. 10C An example table illustrating a benchmark for compute-friendly training is shown according to an example embodiment.

[0032] Fig.11 is a table summarizing example hyperparameters for the LVIS and COCO benchmarks, according to example embodiments.

[0033] Fig.12 An example of an F-VLM feature group according to an example embodiment is shown.

[0034] Fig.13 An example of LVIS new category detection according to an example embodiment is shown.

[0035] Fig.14 An example of Objects365 transfer object detection according to an example embodiment is shown.

[0036] Fig.15 An example of object detection in an indoor scene according to example embodiments is shown.

[0037] Fig.16 An example of object detection in a grocery store scene according to example embodiments is shown.

[0038] Fig.17 is a diagram illustrating the training phase and the inference phase of a machine learning model according to an example embodiment.

[0039] Fig.18 A distributed computing architecture according to an example embodiment is depicted.

[0040] Fig.19 is a block diagram of a computing device according to an example embodiment.

[0041] Fig. 20 Depicted is a network of computing clusters arranged as a cloud-based server system according to an example embodiment.

[0042] Fig.21 is a flowchart of a method according to an example embodiment.

[0043] Fig. 22 is a flow chart of another method according to an example embodiment. DETAILED DESCRIPTION

[0044] In one aspect, the present application relates to an open vocabulary object detection method built on a frozen vision and language model (F-VLM). In another aspect, the present application relates to training an open vocabulary object detection model. Specifically, the present application relates to training only the detector head and combining the detector and VLM outputs of each region at inference time.

[0045] Vision and Language Models (VLMs) have achieved strong open-vocabulary visual recognition capabilities by learning Internet-scale image-text pairs. They are often applied to zero-shot classification (e.g., on ImageNet) with frozen weights without fine-tuning, in stark contrast to the existing paradigm of retraining or fine-tuning when applying VLMs for open-vocabulary detection.

[0046] Zero-shot and open vocabulary recognition is a long-standing problem in computer vision. Some existing methods have used visual attributes to represent categories as binary codebooks and learn to predict the attributes of new categories. Other methods involve using deep learning to learn a joint image-text embedding space. Many studies have shown that there is a bright future for representation learning from natural language associated with images (such as image tags or text descriptions). Recent models have explored large VLMs that are expanded by training billions of image-text pairs and obtain powerful image-text representations through contrastive learning (e.g., using contrastive language-image pre-training (CLIP)). These models achieve high zero-shot performance on many classification benchmarks and show advantages in expanding model capacity.

[0047] While such approaches are based on image-level recognition, object-level understanding may provide a more efficient model. It has been demonstrated that frozen classification models are beneficial for closed vocabulary detection with sufficient detector head capacity. Additionally, frozen VLMs can be used as teacher models and combined with self-training for zero-shot semantic segmentation. However, as described in this paper, frozen VLMs can be used directly as part of an open vocabulary object detector.

[0048] Zero-shot and / or open vocabulary object detection can be expensive and labor-intensive to scale data collection and annotation for large vocabulary detection. Zero-shot detection aims to alleviate the challenge by learning to detect new categories that are not present in the training data. Existing techniques address this problem by aligning image region features with category word embeddings, or by synthesizing visual features using generative models. The Open Vocabulary Detection (OVD) benchmark is introduced to bridge the performance gap between zero-shot detection (ZSD) and supervised learning. Such a model may first be pre-trained on image captioning data to recognize new objects, and then fine-tuned for zero-shot detection.

[0049] Following the OVD benchmark, the Vision and Language Knowledge Distillation (ViLD) model distills the rich representations of a pre-trained VLM into a detector, and Detection Hints (DetPro) also improves on ViLD by applying the idea of ​​hint optimization. RegionCLIP develops a region text pre-training strategy that leverages pre-trained VLMs and image captioning data, while Detector with Image Classes (Detic) jointly trains detectors with weak supervision. Moreover, for example, the Vision and Language Pseudo-Labeling Model VL-PLM explores pseudo-labeling of unlabeled data using object proposals and OVD’s VLM. Another model, Grounded Language-Image Pre-training (GLIP), formulates object detection as a phrase localization task and is pre-trained on various detection, localization, and captioning datasets for zero-shot / few-shot object detection. Similarly, Open World Localized Visual Transformer (OWL-ViT) fine-tunes a pre-trained visual transformer on a range of detection / base datasets. Such approaches are usually based on training the entire detector from scratch, fine-tuning after pre-training for detection, and / or training on a set of detection / base datasets. In contrast, the model described in this paper only trains a standard detector head on a frozen VLM without using any of the aforementioned additional techniques.

[0050] To align image content with text descriptions during training, VLM can learn locally sensitive and discriminative features that are transferable to object detection. Surprisingly, the features of frozen VLM contain rich information that is locally sensitive for describing object shapes and discriminative for region classification. This motivates us to explore the use of frozen VLM features for open vocabulary detection, which requires accurate localization and classification of objects in the wild.

[0051] Overview

[0052] A complete open vocabulary object detection model built on a frozen visual language model (F-VLM) is described. The term "frozen" as used herein generally refers to a state of a neural network where the weights associated with a neural network layer do not change and where the output of a neural network layer is the same across all epochs for a given input. In other words, a frozen neural network can be considered to have been optimized for its respective performance and to remain in such an optimized state. The F-VLM model is configured to scale with the frozen model capacity.

[0053] The overall framework is conceptually simple but effective. Directly using frozen pre-trained vision and language models is much more tractable than performing knowledge distillation and / or weakly supervised learning. Additionally, since the entire language model is frozen, the training cost is significantly lower than other models. Strong quantitative results are shown on the Large-Scale Vocabulary Instance Segmentation (LVIS) dataset and the Common Objects in Context (COCO) dataset. Qualitative cross-dataset generalization results are also shown on a large-scale egocentric dataset (Ego4D).

[0054] During training, the model accesses C B basic category detection labels, but at test time, it needs to be selected from a set of C N To make the setting more practical, we can use a pre-trained Vision and Language Model (VLM) learned from a large number of image-text pairs on the Internet.

[0055] As described in the paper, the model retains the local sensitive features required for downstream detection from the VLM backbone while acting as a strong object classifier. The described techniques reduce training complexity by simplifying the current multi-stage training process. For example, no knowledge distillation or pre-training for detection is required.

[0056] Typically, the underlying VLM may be frozen (and generally referred to as “F-VLM” in this paper), and the detector head may be a trainable component. This results in fewer trainable parameters than competing generalizable models.

[0057] The described F-VLM may result in approximately 200x reduction in computational savings. The technique appears to surpass a state-of-the-art detection benchmark of 6.5 APr on the Large Scale Vocabulary Instance Segmentation (LVIS) dataset and 5.6 on overall mask AP. Moreover, for example, the F-VLM may be significantly faster and cheaper to train. For example, the F-VLM may be trained with very few epochs (e.g., 14.7) and achieve a state-of-the-art APr of 31.0. The described techniques are able to generalize to new categories and new datasets without completely retraining the model, training on a range of detection / base datasets, and / or fine-tuning after pre-training for detection. In some implementations, the detector head may be a Faster R-CNN including a feature pyramid network.

[0058] The model can leverage class-agnostic box regression and mask prediction heads. Thus, for each region proposal, the model can predict a box and a mask for all classes instead of one per class, thereby localizing new objects in an open vocabulary setting. This technique achieves comparable performance to Region-based Language-Image Pretraining (RegionCLIP) on the Common Objects in Context (COCO) open vocabulary object detection benchmark. This technique also achieves comparable performance to state-of-the-art methods for generalizing from the LVIS dataset to the COCO and Objects365 datasets.

[0059] In some embodiments, the detector scores can be combined with the corresponding VLM predictions to obtain an open vocabulary object detection score. In some embodiments, an image encoder can be used for pre-training, and a text encoder can be used to cache text embeddings of the detection dataset vocabulary offline. Also, for example, the last fully connected layer of the described model can include text embeddings for the initial object categories, and can be extended to include new object categories for open vocabulary detection.

[0060] The optimal values ​​of the factors that weigh the relative influence of the initial object categories and the new object categories can be derived. Moreover, the F-VLM trained on one dataset can be directly applied to another dataset by exchanging the vocabulary without any fine-tuning.

[0061] Pre-training of vision and language models

[0062] Figure 1 1 is a diagram illustrating an example training architecture 100 of a neural network according to an example embodiment. An input image 105 may be input into a pre-trained image encoder 110. The pre-trained image encoder 110 may be used as a frozen model. The encoded image may be provided to a trainable detector head 115. The detector head 115 may generate one or more detection boxes and masks 120. The detector head 115 may also embed the image 125 (denoted as r1, ..., r k ) is provided to the detection scoring component 130 to generate a detection score.

[0063] Multiple base categories 135 may be used for training purposes. For example, base categories such as "car", "person", etc. may be provided to a pre-trained text encoder 140. Similar to the image encoder 110, the text encoder 140 may be used as a frozen model to generate text embeddings 145, denoted as t1, ..., t m The text encoder 140 may provide the text embedding 145 to the detection scoring component 130 to generate a detection score. The detection scoring component 130 generates and outputs a detection score for one or more detection boxes and masks 120 based on the image embedding 125 and the text embedding 145, denoted as r i ·tj , where i=1, ..., k and j=1, ..., m. The detection score of one or more loss functions 150 may be evaluated. For example, a box region loss, a box classification loss, and / or a mask classification loss may be determined for training. Legend 155 indicates various types of models used.

[0064] Figure 2 2 is a diagram illustrating an example inference architecture 200 of a neural network according to an example embodiment. An input image 205 may be input into a pre-trained image encoder 210. The pre-trained image encoder 210 may provide the encoded image to a trained detector head 215. The detector head 215 may generate region proposals 220, denoted as b1, ..., b k The detector head 215 may also embed 235 the image (denoted as r1, ..., r k ) is provided to the detection scoring component 240 to generate a detection score. The trained detector head 215 can generate one or more detection boxes and masks 290.

[0065] The region proposal 220 may be provided to a top-level feature map generator 225. In some embodiments, the top-level feature map generator 225 may receive an encoded image from a pre-trained image encoder 210. Additionally, for example, the top-level feature map generator 225 may perform ROI alignment based on the encoded image and the region proposal 220. The output may be provided to a frozen layer of a neural network, a VLM pooling layer 230. The VLM pooling layer 230 is denoted as v1, ..., v k The features 270 are provided to a VLM scoring component 275 to generate a VLM score.

[0066] As about Figure 1 As described, the neural network may have been trained on multiple base categories 245 (e.g., "car", "person", etc.). The pre-trained text encoder 255 can generate text embeddings 260, denoted as t1, ..., t m However, the pre-trained text encoder 255 may be provided with a number of new categories 250 (e.g., “cat,” “ship,” etc.) that were not previously input during training. The text encoder 255 may generate additional text embeddings 265, denoted as t m+1 , ..., t m+n The text encoder 255 can provide the text embedding 260 and the additional text embedding 265 to the detection scoring component 240 to generate a detection score. Based on the image embedding 235, the text embedding 260 and the additional text embedding 265, the detection scoring component 240 generates a detection score, denoted as r i ·t j, where i=1,...,k and j=1,...,m,m+1,...,m+n.

[0067] In some embodiments, VLM scoring component 275 receives text embedding 260 and additional text embedding 265 and generates a VLM score, denoted as v i ·t j , where i=1,...,k and j=1,...,m,m+1,...,m+n.

[0068] In some embodiments, the detection score from detection scoring component 240 can be combined with a VLM score VLM scoring component 275. For example, a geometric mean 280 of the detection score and the VLM score can be determined, and an open vocabulary detection score 285 can be output for the detection box and mask 290.

[0069] These and other aspects of the F-VLM will be described in more detail. Figure 1 and Figure 2 The references can be interchanged because they share common components.

[0070] At test time, F-VLM uses detection box 290 to crop the highest level features 225 of frozen VLM trunk, and calculates VLM score 275 for each region. Trained detector head 215 provides positioning, and classification (e.g., open vocabulary detection score 285) is a combination of detection score 240 and VLM score 275. In some embodiments, open vocabulary object detector can be built on frozen VLM by training detector head only on frozen features, which can ensure that the open vocabulary classification ability of pre-trained VLM is retained. At test time, we combine detection score 240 with VLM score 275 to obtain open vocabulary object detection score 285. By directly using frozen pre-trained models (e.g., image encoder 210, text encoder 255, VLM pooling 230), the method is simple and easy to expand.

[0071] refer to Figure 1 ,When training, F-VLM is a standard detector where the last classification layer of the neural network is replaced with text embeddings 145 from the base class 135. The detector head 115 can be trained while the remaining model can be frozen.

[0072] Vision and Language Models (VLMs) are popular because of their rich knowledge and strong representation of vision and language domains. Using a frozen VLM allows the neural network to retain such knowledge as much as possible, thereby minimizing the workload and / or cost of adapting the VLM to open vocabulary detection. For illustrative purposes, a contrastive pre-trained VLM is described. Contrastive VLMs typically train image and text encoders jointly with contrastive targets. Contrastive VLMs are easy to use for detection and / or segmentation tasks and have been adopted by existing open vocabulary detection and / or segmentation models. The frozen image encoder 110 can be used as a detector backbone, and the frozen text encoder 140 is used to cache text embeddings of the vocabulary of the detection dataset offline.

[0073] In some embodiments, the VLM image encoder 110 may include two parts: 1) Feature Extractor Such as, for example, ResNet-50, and 2) the last feature pooling layer Such as, for example, an attention pooling layer. Can be used with image feature extractors The same backbone architecture, which allows the use of frozen weights directly and allows the inheritance of rich semantic knowledge. Together with the backbone initialization, the same image preprocessing scheme as VLM pre-training can be used to preserve open vocabulary recognition capabilities. The last VLM pooling layer (For example, Figure 2 The VLM pooling 230) can be used for open vocabulary recognition at test time. Based on the frozen backbone features, the Mask R-CNN head can be used as the detection head, and the Feature Pyramid Network is used as the detector head. The detector head can be randomly initialized and may be the only trainable component of the F-VLM, such as Figure 1 Despite image-level pre-training, the frozen VLM backbone appears to include sufficient local-sensitive features for accurate downstream detection.

[0074] For example, to understand the effectiveness of F-VLM, k-means clustering can be performed to detect the structure present in frozen VLM features (e.g., CLIP). In some embodiments, the CLIPR50×4 backbone and LVIS datasets can be used for visualization. In general, the last layer output features can be used for clustering because these features can also be used for zero-shot region classification. Figure 7 We show that features form clusters around salient objects of a scene (e.g., skis, motorcycles, people) and naturally separate object parts (e.g., donut toppings, bus wheels) without explicit supervision.

[0075] Text Embedding Region Classifier

[0076] For a more precise description, the input image 105 can be represented as I, and the backbone features from the image encoder can be represented as from Generate region embedding r b The function can be expressed as And a given box region proposal can be denoted as b. In some embodiments, the box region proposal may involve FPN, ROI-Align, and Faster R-CNN heads. Accordingly,

[0077]

[0078] Standard detectors typically use a K-way classifier because the training and test time categories are the same. However, such a design does not support an open vocabulary setting where new categories may be added at test time. To accommodate this, the last fully connected layer can be replaced with a text embedding 145 of the base category 135 (see Figure 1 ). At inference time, the text embedding can then be expanded to include text embeddings 260 of the base categories 245 and additional text embeddings 265 of the new categories 250 for open vocabulary detection (see Figure 2 ). The advantage of this design is that the system can be generalized to the embedding space close to C B New category.

[0079] To generate text embeddings, it may be necessary to use the matching text encoder 140 (respectively text encoder 265) of the image encoder 110 (respectively image encoder 210) since they may have been pre-trained jointly. B In addition, the background category can be represented by a generic phrase “background” to be compatible with other categories. B Region proposals 220 that do not match the ground truth boxes in may be considered as background. For each region, we can determine r b With C B The cosine similarity of the text embedding and the “context” of , and a learnable temperature τ can be applied on the logit. The detection score z(r b ) can be determined as:

[0080]

[0081] where cos(a, b) = a T b / (||a||||b||) and t i represents the text embedding of class i. A standard softmax cross entropy loss can be applied on the logits (see Figure 1 ). At test time, a “background” class can be maintained and the text embeddings can be moved from C B Extension to C B ∪CN To include basic categories 245 and new categories 250 for open vocabulary detection.

[0082] Open vocabulary recognition

[0083] The ability to perform open vocabulary classification at the region level is another advantage of F-VLM. Since the backbone features are frozen, they generally do not overfit to the base class and can be directly pruned for region-level classification. F-VLM performs this open vocabulary classification at test time (e.g., Figure 2 The highest level feature map 225).

[0084] In order to obtain the features of region b, the VLM pooling layer 230 (denoted as ) is applied to the cropped backbone output features Since the VLM pooling layer 230 receives a fixed-size input, such as, for example, 7×7 for R50, ROI-Align can be used. Crop and resize regional features (see Figure 2 The top-level feature map 225 of the detector head is trained in one stage. This reduces complexity and can be more space-efficient. In addition, with The VLM region features of are not pruned during training because the backbone features are frozen. Usually, the VLM region embedding v b It can be determined as:

[0085]

[0086] Where b represents the box area, and v b Corresponds to Figure 2 In the representation, v1, ..., v k Features 270. As mentioned above, It is used at test time, not during training. Similar to Equation 2 for the detection score, the VLM score can be determined by the cosine similarity as follows:

[0087]

[0088] where T is a fixed temperature parameter and the text embedding includes C at inference time B and C N (See Figure 2245 and the new category 250). In some embodiments, a fixed temperature parameter may be used to adjust the ratio of the VLM score 275 relative to the detection score 240 in Equation 2. In the special case where region b is equal to the entire input image 205, denoted as w(v b )’s VLM score 275 becomes equal to the zero-sample image classification score.

[0089] Although no training was done on the area, The cropped region features maintain high open vocabulary recognition ability. However, in some embodiments, the cropped region features may not be sensitive enough to the localization quality of the region. For example, loosely located boxes and tightly located boxes may both include similar features. While this may be beneficial for classification, it may pose a challenge for detection because it is desirable that the detection score also reflects the localization quality. To address this issue, for each region b and category i, the VLM score 275 (denoted as w(v) in Equation 4) may be subtracted by applying a geometric mean 280. b ) i ) and the detection score 240 (expressed as z(r in Equation 2) b ) i ) combination. It is expressed as s(r b ) i An open vocabulary detection score of 285 can be determined as:

[0090]

[0091] where α, β∈[0,1] are adjustable parameters controlling the weights of the VLM scores for the base class and / or the new class, and the background score can be directly received from the detector head 215 to obtain s(r b )0=z(r b )0. Compared with the integrated system, the F-VLM described in this paper is simple in design and does not rely on knowledge distillation and / or dual Faster R-CNN heads.

[0092] Open Vocabulary Positioning

[0093] Locating new objects and separating them from the background is a challenging problem in open vocabulary detection. Standard detectors are typically not designed to localize new objects because such models apply class-specific localization, including box regression and mask prediction heads, such as, for example, Mask R-CNN. As described herein, class-independent box regression and mask prediction heads may be used instead. In some embodiments, for each region proposal 220, a box and a mask may be predicted for the base class and the new class, rather than a box and a mask for each class. This simple change enables localization of new objects in an open vocabulary setting. In general, the F-VLM framework may not be specific to the choice of the Mask R-CNN detector head, and other models may also be applied.

[0094] Experiment: Example implementation details

[0095] For ease of illustration, the detector head 115 (and accordingly the detector head 215) can be selected as a Mask R-CNN with a feature pyramid network. In some embodiments, the model can be trained for 46.1k iterations using 1024×1024 image size, large-scale jittering, a batch size of 256, a weight decay of 1e-4, a momentum of 0.9, and an initial learning rate of 0.36. For the score combination, α=0.35 and β=0.65 in Equation 5. A maximum of 300 detections can be used per image, and the temperature T=0.01 is set in Equation 4. Moreover, for example, the CLIP hint template can be used with the average text embedding for each category.

[0096] Experiment: Open Vocabulary Detection Benchmark

[0097] LVIS Benchmark

[0098] The model can be evaluated on the LVIS dataset, which includes a large and diverse set of 1903 object categories suitable for open vocabulary detection. In some embodiments, frequent and common categories can be used as the base categories C for training. B , and the rare category can be used as a new category C for testing N The benchmark can be based on the metric mask AP r To ensure reproducibility, the average of five (5) independent runs is determined. For fair comparison, the same Mask R-CNN head architecture used in the existing model is used, and a similar large-scale jittering recipe is used.

[0099] Figure 3300 is a table showing a comparison between various object detection models according to an example embodiment. On new categories, F-VLM's mask average precision (AP) is 6.5 higher than the best existing method. The method may use the same instance-level supervision from LVIS base categories, CLIP pre-training, and fixed hint templates. In some embodiments, hint optimization can be used to improve the fixed hint template. Figure 3 Results for LVIS are presented. In an R50 comparison, F-VLM ranks second among alternatives based on knowledge distillation, pre-training, and / or weakly supervised joint training. The DetPro model shows the effectiveness of fast optimization, which in some embodiments can be applied to F-VLM. In a system-level comparison, the performance of F-VLM scales with increasing frozen model capacity, although the number of trainable parameters may remain fixed. An experimental F-VLM model can achieve 32.8 AP r , which is +14.2AP different from the R50 baseline r , which appears to be a significant improvement over known results on this benchmark. On new categories, the mask AP of the F-VLM model is r 6.5 higher than ViLD-EN-B7 (and +5.6 higher in overall mask AP).

[0100] COCO Benchmark

[0101] There is a lot of research on zero-shot detection and open vocabulary detection benchmarks on COCO. The COCO vocabulary can be divided into 48 basic categories for training and 17 new categories for testing. Results are reported in a generalized detection setting without instance segmentation. The metric AP50 for new categories can be used. Similar to LVIS, the average of five (5) independent runs can be calculated to ensure reproducibility.

[0102] Due to the small number of training categories, there may be a tendency to overfitting when the same LVIS training recipe is reused. F-VLM does not rely on additional objectives such as knowledge distillation or weak supervision to balance overfitting. Therefore, to mitigate this, in some embodiments, the training epochs and background weights can be reduced, and weight decay can be increased.

[0103] Figure 44 is a table 400 showing a comparison between various object detection models according to an example embodiment. As shown, F-VLM is competitive with other methods trained using various sources. Methods use ResNet50 backbone. RegionCLIP is additionally pre-trained using COCO Captions or CC3M. F-VLM uses frozen CLIP backbone directly. F-VLM appears to significantly outperform RegionCLIP's CLIP-R50 pre-trained version that does not utilize pre-training on caption data. Compared to other methods, F-VLM appears to provide better performance without using pre-training for detection, weakly supervised learning, and / or knowledge distillation.

[0104] Fig.11 is a table 1100 summarizing example hyperparameters for the LVIS and COCO benchmarks according to an example embodiment. On LVIS, the same hyperparameters as the existing model may be employed, except for a shorter timeline (due to frozen backbone) and background weights. Hyperparameter differences for COCO may be introduced to mitigate overfitting on the ZSD-COCO split of 48 categories, which is significantly smaller than the 800 LVIS base categories. F-VLM does not use other objectives such as, for example, knowledge distillation or weak supervision to balance overfitting, so hyperparameter differences may be acceptable.

[0105] Some differences in the optimal hyperparameters for different backbone architectures can be observed. Using an R50×64 backbone, one can expect 1.0 AP when T=0.02 compared to the default T=0.01. r For the R50 backbone, when applying gradient clipping with a maximum gradient norm of 1.0, a 0.5AP improvement is observed compared to no gradient clipping. r improvements.

[0106] Training Resource Benchmark

[0107] Freezing the VLM can also help save training resources. This benchmark may be based on the ViLD model as it is generally comparable to the F-VLM. Both models employ the same Mask R-CNN head configuration and training recipe, and neither model relies on pre-training for detection. For ViLD, the training cost of a TPUv3 core with the same batch size can be compared. Data on ViLD training time and resource usage is publicly available. To keep the benchmark simple, a pre-trained VLM is assumed and its training cost can be excluded from the comparison. For the F-VLM, an R50×64 backbone can be used and the average of five (5) independent runs can be determined.

[0108] Figure 6 is a table 600 showing example results of performance and training cost tradeoffs according to an example embodiment.r values ​​to demonstrate the tradeoff between performance and training cost. Table 600 shows that F-VLM can achieve the best performance with less computation. Compared to the system-level ViLD-EN-B7 model, F-VLM appears to achieve better performance in 7.4 training epochs, is 226 times more computationally efficient, and has 57 times less wall clock time. The efficiency improvement generally comes from freezing the backbone, which greatly simplifies the learning process. In addition to saving resources, F-VLM also saves a lot of memory at training time by running the backbone in inference mode. In some embodiments, the F-VLM system runs almost as fast as a standard detector at inference time because the only addition is a single attention pooling layer on the detected region features (see Figure 2 ).

[0109] Metastasis Detection Benchmark

[0110] F-VLM can be evaluated as a generic detector for different data sources, with a view to moving towards non-dataset-specific detection. F-VLM trained on one dataset can be directly applied to another dataset by exchanging the vocabulary without any fine-tuning, such as, for example, replacing 1903 LVIS categories with COCO80 categories. The model can be trained on LVIS base categories and tested on COCO and Objects365-v1 validation splits after the transfer setting of ViLD. Since the vocabulary of COCO and Objects365 is smaller than that of LVIS, it may be difficult to avoid category and image overlap. The vocabulary overlap between COCO / Objects365 and LVIS base categories is shown to be 91% and 63%, respectively.

[0111] Figure 55 is a table 500 showing example results on various test datasets according to example embodiments. For example, the generalization ability of a detector trained on LVIS using F-VLM can be evaluated on COCO and Object365 datasets. The results reported in Box AP are the average of 5 runs. The performance of F-VLM appears to improve steadily as the frozen model capacity is expanded. On Objects365 / COCO, F-VLM is +3.2 / +5.9 higher than the existing work ViLD and +4.9 / +5.6 higher than DetPro, thus narrowing the gap with supervised models on COCO (-33%) and Objects365 (-40%). There does not seem to be a distinction between base and new categories in the transfer setting, so all categories can be considered new categories. Moreover, for example, in Equation 5, β can be used alone to combine the detection and VLM scores. Moreover, for COCO, it seems that only the detection score (β=0) is needed, while the best detection score on Objects365 is β=0.3 to 0.4. F-VLM can show strong scaling properties on COCO / Objects365 by increasing the backbone capacity, with a gain of +7.3 / +5.8AP.

[0112] Ablation

[0113] Ablation studies on backbone fine-tuning, score fusion design / parameters, feature pyramid capacity, and background weights are provided below.

[0114] Fine-tuning

[0115] In an exploration of fine-tuning versus freezing the backbone (see Table 800A), it appears that fine-tuning improves standard detection (basic categories) but may slightly degrade open word detection (novel categories).

[0116] Fig. 8A 800A is a table showing example results of VLM score weights for new classes according to an example embodiment. Results showing advantages and disadvantages of backbone fine-tuning compared to frozen backbone. In general, fine-tuning the backbone using the same training recipe may diverge, and gradient clipping (max norm = 1.0) may be applied to significantly reduce the backbone learning rate. Table 800A shows that while fine-tuning can benefit base classes, it may slightly hurt new classes that have a higher memory and / or computational footprint.

[0117] Score Fusion

[0118] Figure 8B is a table 800B showing example results of fusing VLM and detection scores using arithmetic mean and geometric mean, according to an example embodiment. In the score fusion study (see Table 800B), the geometric mean appears to be significantly better than the arithmetic mean (+8AP r). This may be because the geometric mean requires high-scoring regions to have both good detection and VLM scores, while the arithmetic mean may favor regions with high detection or VLM scores. A dense grid scan of α, β confirms that the 8-point gap between the geometric mean and the arithmetic mean still holds. A deeper investigation of the score fusion parameters can be performed and the results are shown in 9A to 9C From the table we can see that β is the main adjustable parameter of our model and the performance is relatively robust to α. For most practical use cases, it is recommended to set T = 0.01. The temperature τ in Equation 2 can be learned automatically and does not need to be adjusted.

[0119] 9A to 9C Tables 900A-C showing fractional fusion parameters according to an example embodiment are shown. In Tables 900A and 900B, the results of the fractional fusion weights are shown. It seems to be observed that β=0.65 and α=0.35 may be beneficial parameter values. Neither the detection score nor the VLM score alone are sufficient, as β=0, 1 both produce optimal performance. In Table 900C, the temperature in Equation 4 is investigated and the optimal value seems to be T=10 -2 , which is usually less than the value of the learnable parameter τ≈1.0 at the end of training (see Equation 2). This generally emphasizes the need to use a separate T for the VLM score, rather than using τ for both the detection score and the VLM score.

[0120] Fig. 10A An example table 1000A showing the capacity of a feature pyramid is shown according to an example embodiment. In Table 1000A, the results of increasing the capacity of a feature pyramid to enhance the effect of learning representations on frozen backbone features are shown. A larger feature pyramid appears to improve the basic category (AP c ,AP f ), without compromising the new category (AP r ). In general, larger pyramids benefit standard detection (basic categories) without hurting open vocabulary detection (new categories).

[0121] Fig. 10B An example table 1000B showing the effect of background weight on open vocabulary detection is shown according to an example embodiment. In Table 1000B, the effect of background weight on open vocabulary detection is shown, and the background weight appears to have some benefit (e.g., 0.1 to 0.5 AP r ). For example, Table 1000B shows that a background weight of 0.9 is slightly better than the default value of 1.0. Therefore, a background weight of 0.9 may be used as the default value. The results may be the average of three (3) independent runs.

[0122] Detection visualization

[0123] Figure 7 An example of an object detection task on various images 700 according to an example embodiment is shown. The first column C1 and the second column C2 show images of open vocabulary detection on the LVIS new category. The detected new categories include a fedora, a martini, a pennant, and a football helmet. The third column C3 and the fourth column C4 show images of transfer detection on Objects365. The detected new categories include a camel, a slide, and a goldfish. The fifth column C5 and the sixth column C6 show images of transfer detection on the real-world egocentric application Ego4D. The detected new categories include an exit sign, a recycling bin, a window, soy sauce, a wooden basket, cereal, a bag of cookies, instant noodles, salad dressing, and ketchup. Despite the large domain shift, F-VLM is still able to detect many new and common objects.

[0124] Computationally friendly training

[0125] To facilitate comparison with the broader research community, the efficacy of F-VLM is validated in more computationally friendly 1× (12 epochs) and 3× (36 epochs) settings by using smaller batch sizes and without large-scale jitter (LSJ) augmentation.

[0126] Fig. 10C An example table 1000C showing a benchmark for computationally friendly training according to an example embodiment is shown. Table 1000C shows that by utilizing frozen backbone, F-VLM is robust to shorter schedules and smaller batch sizes. All results are reported as an average of five (5) runs.

[0127] F-VLM appears to be robust to the number of training epochs for the smallest and largest backbones, and to batch size (with or without LSJ). This is in contrast to the sensitivity of fully supervised learning to these hyperparameters, and is consistent with Figure 6 The 600 frozen backbone helps with training efficiency and stability of the results.

[0128] Freeze the structure of a feature

[0129] To understand the effectiveness of F-VLM, k-means clustering can be performed to detect the structure present in frozen VLM features (e.g., CLIP). CLIP R50×4 backbone and LVIS datasets can be used for visualization. In some embodiments, the last layer output features can be used for clustering, because these features can also be used for zero-shot region classification.

[0130] Fig.12An example of a F-VLM feature grouping according to an example embodiment is shown. Salient objects and object parts naturally emerge from clustering of frozen VLM features. As shown, the features appear to form clusters around salient objects of the scene (e.g., skis, motorcycles, people) and naturally separate object parts (e.g., donut toppings, bus wheels) without explicit supervision.

[0131] Metastasis Detection Benchmark

[0132] Simple name matching shows overlap, while removing near-duplicates (e.g. synonyms and non-alphabetic character removal) shows more overlap. Also, COCO appears to have more overlap than Objects365 due to its smaller vocabulary.

[0133] Memory usage

[0134] The memory consumption of F-VLM is comparable to that of Mask R-CNN, where the class-specific heads become class-independent. In addition, F-VLM has significant memory savings potential compared to existing methods that fine-tune the backbone (especially in the case of large backbones). During training, F-VLM does not need to store forward activations, gradients, or gradient moments, and the memory usage of the backbone is only the backbone weights and a small amount of current activations. This makes F-VLM highly memory efficient, especially in the case of large backbones. In practice, the actual memory usage depends on the low-level implementation of each deep learning library.

[0135] Alternative Example VLM

[0136] As described herein, CLIP is used to develop a simple open vocabulary detection recipe based on a frozen backbone. Additional and / or alternative pre-trained VL models, such as multimodal encoders, subtitle models, masked image / language models, can be used. For example, pre-trained VLMs based on ViT can be used. These VLMs require a single-scale detector based on ViT (such as ViTDet) to adapt them to open vocabulary detection. In order to use VLMs like ALBEF and BLIP, it can be determined that the use of a multimodal encoder (compared to a dual encoder) is effective in determining all pairs of region-text similarities.

[0137] Fig.13 An example of LVIS new category detection according to an example embodiment is shown. Although it is simple to use the frozen VLM, the F-VLM can detect many new categories. The white arrows point to new objects that the F-VLM correctly detected.

[0138] Fig.14 An example of Objects365 transfer object detection according to an example embodiment is shown. F-VLM can be applied to new datasets and detect many challenging categories without further fine-tuning.

[0139] Applications of egocentric data

[0140] One of the advantages of open vocabulary detection is to test out-of-distribution data using categories dynamically given by the user. Therefore, F-VLM can be applied to real-world egocentric applications Ego4D. F-VLM can be trained on a mixture of the full LVIS, Objects365, and COCO datasets to expand the trained vocabulary for in-the-wild applications, and can use an R50×16 backbone. To evaluate the model for transfer detection, the model may not be trained on Ego4D. Users can provide categories based on visual inspection of the video.

[0141] In general, F-VLM may be able to detect many objects in egocentric videos despite large domain shifts and challenging viewing conditions. Specifically, F-VLM may be able to detect new categories not present in the training set, such as light switches, lamps, door locks, sauces and seasonings, a bag of candy, canned goods, and burritos, as shown below: Fig.15 and Fig.16 shown.

[0142] Fig.15 An example of object detection in an indoor scene according to an example embodiment is shown. As shown, F-VLM detects many objects in the scene even under challenging viewpoints, occlusions, and lighting conditions. For the indoor scene, the user-provided category names are as follows: dishes, cabinets, stoves, towels, cleaning rags, ventilators, knobs, sauces and dressings, steel lids, windows, blinds, plants, light switches, lights, doors, carpets, exit signs, doormats, hair, door locks, trees, posters on the wall, stickers on the wall, faucets, recycling bins, shelves, hands, cans, cartons, trash cans, Christmas trees, plastic containers, refrigerators.

[0143] Fig.16 An example of object detection in a grocery store scene according to an example embodiment is shown. The scene is very crowded with a variety of objects. However, F-VLM is able to detect many objects. For the grocery store scene, the category names provided by the user are as follows: exit sign, poster, chocolate bar, bag of candy, bag of cookies, snacks, Oreos, soy sauce, apples, pears, oranges, grapes, price tags, cereal, instant noodles / ramen, cookies, ATM machine, instant noodles, wooden basket, red ramen bowl, magazine, medicine, mayonnaise, ketchup, cup noodles, burritos, Lays / Sun chips, sauce, black paper box, salad dressing, canned food.

[0144] Sample Application

[0145] In one aspect, the input data for object detection includes image data. The accuracy of object detection performed by the F-VLM-based detector head is generally higher than the accuracy of other models. The models described herein can perform open vocabulary object detection and can be used to detect a wide variety of objects and accurately capture many new objects. For example, the F-VLM can detect many long-tail categories without fine-tuning.

[0146] In some embodiments, pre-trained vision and language models can be acquired, and detector heads can be added and trained for object detection. Moreover, for example, the model can be used to localize new objects in an open vocabulary setting. As another example, a F-VLM trained on one dataset can be directly applied to another dataset by exchanging the vocabulary without any fine-tuning.

[0147] In some embodiments, the model can reside on the mobile device and perform object detection. Also, for example, the model can analyze video data instead of image data for open tracking (tracking objects as they enter / leave the frame and / or specific locations within the frame), anomaly detection (for security / surveillance tracking), and / or segmentation. Detected categories can be linked to features of other datasets, including location, contact, date, and / or search.

[0148] The technology can be applied to object detection in robotic vision, autonomous and semi-autonomous driving, smart cameras, security and / or surveillance cameras, satellite image processing, space exploration, etc. Moreover, for example, in autonomous and semi-autonomous driving applications, the technology can be applied to object detection during point cloud processing of videos and still images.

[0149] As another example, the F-VLM including open vocabulary detection can be applied to product images. For example, the F-VLM model can be applied to shopping images (e.g., product-centric) and camera images (e.g., from mobile platforms).

[0150] These and other example applications are contemplated to be within the scope of the present disclosure.

[0151] Train machine learning models to generate inferences / predictions

[0152] Fig.17 A diagram 1700 is shown illustrating a training phase 1702 and an inference phase 1704 of a trained machine learning model 1732, according to an example embodiment. Some machine learning techniques involve training one or more machine learning algorithms on an input set of training data to identify patterns in the training data and provide output inferences and / or predictions about (patterns in) the training data. The resulting trained machine learning algorithm may be referred to as a trained machine learning model. For example, Fig.17A training phase 1702 is shown, where one or more machine learning algorithms 1720 are being trained on training data 1710 to become a trained machine learning model 1732. Then, during an inference phase 1704, the trained machine learning model 1732 may receive input data 1730 and one or more inference / prediction requests 1740 (perhaps as part of the input data 1730), and responsively provide one or more inferences and / or predictions 1750 as output.

[0153] Therefore, the trained machine learning model 1732 may include one or more models of one or more machine learning algorithms 1720. The machine learning algorithm 1720 may include, but is not limited to: an artificial neural network (e.g., a convolutional neural network, a recurrent neural network, a Bayesian network, a hidden Markov model, a Markov decision process, a logistic regression function, a support vector machine, a suitable statistical machine learning algorithm, and / or a heuristic machine learning system as described herein). The machine learning algorithm 1720 may be supervised or unsupervised, and may implement any suitable combination of online learning and offline learning.

[0154] In some examples, the machine learning algorithm 1720 and / or the trained machine learning model 1732 can be accelerated using an on-device coprocessor, such as a graphics processing unit (GPU), a tensor processing unit (TPU), a digital signal processor (DSP), and / or an application-specific integrated circuit (ASIC). Such an on-device coprocessor can be used to accelerate the machine learning algorithm 1720 and / or the trained machine learning model 1732. In some examples, the trained machine learning model 1732 can be trained, resident, and executed to provide reasoning on a specific computing device, and / or can otherwise make reasoning for a specific computing device.

[0155] During the training phase 1702, the machine learning algorithm 1720 may be trained using unsupervised, supervised, semi-supervised, and / or reinforcement learning techniques by providing at least the training data 1710 as training input. Unsupervised learning involves providing a portion (or all) of the training data 1710 to the machine learning algorithm 1720, and the machine learning algorithm 1720 determining one or more output inferences based on the portion (or all) of the provided training data 1710. Supervised learning involves providing a portion of the training data 1710 to the machine learning algorithm 1720, wherein the machine learning algorithm 1720 determines one or more output inferences based on the portion of the provided training data 1710, and accepting or correcting the output inferences based on correct results associated with the training data 1710. In some examples, supervised learning of the machine learning algorithm 1720 may be governed by a set of rules and / or a set of labels for the training input, and the set of rules and / or the set of labels may be used to correct the inferences of the machine learning algorithm 1720.

[0156] Semi-supervised learning involves obtaining the correct result for a portion (but not all) of the training data 1710. During semi-supervised learning, supervised learning is used for a portion of the training data 1710 that has the correct result, and unsupervised learning is used for a portion of the training data 1710 that does not have the correct result. Reinforcement learning involves the machine learning algorithm 1720 receiving a reward signal about a previous reasoning, where the reward signal can be a numerical value. During reinforcement learning, the machine learning algorithm 1720 can output an inference and receive a reward signal in response, where the machine learning algorithm 1720 is configured to try to maximize the numerical value of the reward signal. In some examples, reinforcement learning also utilizes a value function that provides a numerical value representing the expected sum of the numerical values ​​provided by the reward signal over time. In some examples, the machine learning algorithm 1720 and / or the trained machine learning model 1732 can be trained using other machine learning techniques, including but not limited to incremental learning and curriculum learning.

[0157] In some examples, the machine learning algorithm 1720 and / or the trained machine learning model 1732 may use transfer learning techniques. For example, the transfer learning technique may involve the trained machine learning model 1732 being pre-trained on a set of data and additionally trained using the training data 1710. More specifically, the machine learning algorithm 1720 may be pre-trained on data from one or more computing devices, and the resulting trained machine learning model is provided to the computing device CD1, where CD1 intends to execute the trained machine learning model during the inference phase 1704. Then, during the training phase 1702, the pre-trained machine learning model may be additionally trained using the training data 1710, where the training data 1710 may be derived from the kernel data and non-kernel data of the computing device CD1. Such further training of the machine learning algorithm 1720 and / or the pre-trained machine learning model using the training data 1710 of the CD1 data may be performed using supervised learning or unsupervised learning. Once the machine learning algorithm 1720 and / or the pre-trained machine learning model has been trained on at least the training data 1710, the training phase 1702 may be completed. The resulting trained machine learning model can be used as at least one of the trained machine learning models 1732.

[0158] Specifically, once the training phase 1702 has been completed, the trained machine learning model 1732 can be provided to the computing device (if not already on the computing device). After the trained machine learning model 1732 is provided to the computing device CD1, the reasoning phase 1704 can begin.

[0159] During the inference phase 1704, the trained machine learning model 1732 may receive input data 1730 and generate and output one or more corresponding inferences and / or predictions 1750 about the input data 1730. Thus, the input data 1730 may be used as an input to the trained machine learning model 1732 for providing corresponding inferences and / or predictions 1750 to kernel components and non-kernel components. For example, the trained machine learning model 1732 may generate inferences and / or predictions 1750 in response to one or more inference / prediction requests 1740. In some examples, the trained machine learning model 1732 may be executed by a portion of other software. For example, the trained machine learning model 1732 may be executed by an inference or prediction daemon so that it is readily available upon request for providing inferences and / or predictions. The input data 1730 may include data from the computing device CD1 executing the trained machine learning model 1732 and / or input data from one or more computing devices other than CD1. For example, the input data 1730 may include a collection of images provided by one or more sources, and text labels of objects in the images. Other types of input data are also possible.

[0160] Inferences and / or predictions 1750 may include output images, output intermediate images, numerical values, and / or other output data produced by a trained machine learning model 1732 operating on input data 1730 (and training data 1710). In some examples, the trained machine learning model 1732 may use the output inferences and / or predictions 1750 as input feedback 1160. The trained machine learning model 1732 may also rely on past inferences as input for generating new inferences.

[0161] A neural network including a frozen VLM and a trainable detector head may be an example of a machine learning algorithm 1720. After training, the trained version of the neural network may be an example of a trained machine learning model 1732. In this approach, an example of one or more inference / prediction requests 1740 may be a request to predict an object in an input image, and a corresponding example of an inference and / or prediction 1750 may be a predicted object in the input image.

[0162] In some examples, a computing device CD_SOLO may include a trained version of a neural network based on a frozen VLM (perhaps after training). The computing device CD_SOLO may then receive a request to predict an object in an input image and use the trained version of the neural network to predict the object.

[0163] In some examples, two or more computing devices CD_CLI and CD_SRV may be used to provide an output image; for example, a first computing device CD_CLI may generate a request to predict an object in an input image and send the request to a second computing device CD_SRV. CD_SRV may then use a trained version of a neural network to predict an object and respond to CD_CLI's request for the predicted object. CD_CLI may then provide the requested predicted object upon receiving a response to the request (e.g., using a user interface and / or display, a printed copy, an electronic communication, etc.).

[0164] Sample Data Network

[0165] Fig.18 A distributed computing architecture 1800 is depicted according to an example embodiment. The distributed computing architecture 1800 includes server devices 1808, 1810 configured to communicate with programmable devices 1804a, 1804b, 1804c, 1804d, 1804e via a network 1806. The network 1806 may correspond to a local area network (LAN), a wide area network (WAN), a WLAN, a WWAN, an intranet, the public Internet, or any other type of network configured to provide a communication path between networked computing devices. The network 1806 may also correspond to a combination of one or more LANs, WANs, intranets, and / or the public Internet.

[0166] although Fig.18 Only five programmable devices are shown, but the distributed application architecture can serve tens, hundreds, or thousands of programmable devices. In addition, programmable devices 1804a, 1804b, 1804c, 1804d, 1804e (or any additional programmable devices) can be any kind of computing device, such as a mobile computing device, a desktop computer, a wearable computing device, a head-mounted device (HMD), a network terminal, a mobile computing device, and the like. In some examples, such as shown in programmable devices 1804a, 1804b, 1804c, 1804e, the programmable device can be directly connected to the network 1806. In other examples, such as shown in programmable device 1804d, the programmable device can be indirectly connected to the network 1806 via an associated computing device (such as programmable device 1804c). In this example, programmable device 1804c can act as an associated computing device for transmitting electronic communications between programmable device 1804d and network 1806. In other examples, such as shown by programmable device 1804e, the computing device may be part of and / or located within a vehicle (such as a car, truck, bus, boat or ship, airplane, etc.). Fig.18 In other examples not shown, a programmable device may be connected to network 1806 either directly or indirectly.

[0167] The server devices 1808, 1810 may be configured to perform one or more services as requested by the programmable devices 1804a-1804e. For example, the server devices 1808 and / or 1810 may provide content to the programmable devices 1804a-1804e. The content may include, but is not limited to, web pages, hypertext, scripts, binary data (such as compiled software), images, audio, and / or video. The content may include compressed and / or uncompressed content. The content may be encrypted and / or unencrypted. Other types of content are also possible.

[0168] As another example, server devices 1808 and / or 1810 may provide programmable devices 1804a-1804e with access to software for database, search, computing, graphics, audio, video, web / internet utilization, and / or other functions. Many other examples of server devices are possible.

[0169] Computing device architecture

[0170] Fig.19 is a block diagram of an example computing device 1900 according to an example embodiment. Specifically, Fig.19 The computing device 1900 shown in may be configured to perform at least one function, and / or method 1400 of and / or related to open vocabulary object detection based on frozen visual and language models.

[0171] The computing device 1900 may include a user interface module 1901, a network communication module 1902, one or more processors 1903, a data storage device 1904, one or more cameras 1912, one or more sensors 1914 and a power system 1916, all of which may be linked together via a system bus, network or other connection mechanism 1905.

[0172] The user interface module 1901 may be operable to send data to and / or receive data from an external user input / output device. For example, the user interface module 1901 may be configured to send data to and / or receive data from a user input device such as a touch screen, a computer mouse, a keyboard, a keypad, a touchpad, a trackball, a joystick, a voice recognition module, and / or other similar devices. The user interface module 1901 may also be configured to provide output to a user display device such as one or more cathode ray tubes (CRTs), liquid crystal displays, light emitting diodes (LEDs), displays using digital light processing (DLP) technology, printers, light bulbs, and / or other similar devices, whether now known or later developed. The user interface module 1901 may also be configured to generate sound output using devices such as speakers, speaker jacks, audio output ports, audio output devices, headphones, and / or other similar devices. The user interface module 1901 may also be configured with one or more tactile devices that can generate tactile outputs such as vibrations and / or other outputs that can be detected by touch and / or physical contact with the computing device 1900. In some examples, user interface module 1901 may be used to provide a graphical user interface (GUI) for utilizing computing device 1900 , such as, for example, a graphical user interface of a mobile telephone device.

[0173] The network communication module 1902 may include one or more devices that provide one or more wireless interfaces 1907 and / or one or more wired interfaces 1908 that may be configured to communicate via a network. The wireless interface 1907 may include one or more wireless transmitters, receivers, and / or transceivers, such as Bluetooth. TM Transceiver, Transceiver, Wi-Fi TM Transceiver, WiMAX TM Transceiver, LTE TM The wired interface 1908 may include one or more wired transmitters, receivers, and / or transceivers, such as an Ethernet transceiver, a Universal Serial Bus (USB) transceiver, or a similar transceiver that can be configured to communicate via a twisted pair, a coaxial cable, a fiber optic link, or a similar physical connection to a wired network.

[0174] In some examples, the network communication module 1902 can be configured to provide reliable, protected and / or authenticated communications. For each communication described herein, information for facilitating reliable communications (e.g., ensuring message delivery) can be provided, which may be part of a message header and / or message trailer (e.g., packet / message ordering information, encapsulation header and / or encapsulation trailer, size / time information, and transmission verification information (such as cyclic redundancy check (CRC) and / or parity check value)). One or more encryption protocols and / or algorithms can be used to ensure the security of communications (e.g., encoding or encryption) and / or decrypt / decode communications, such as, but not limited to, the Data Encryption Standard (DES), the Advanced Encryption Standard (AES), the Rivest-Shamir-Adelman (RSA) algorithm, the Diffie-Hellman algorithm, a secure socket protocol (such as, a secure socket layer (SSL) or a transport layer security (TLS)), and / or a digital signature algorithm (DSA). In addition to those protocols and / or algorithms listed herein, other cryptographic protocols and / or algorithms can also be used to protect communications (and then decrypt / decode them).

[0175] The one or more processors 1903 may include one or more general purpose processors and / or one or more special purpose processors (e.g., digital signal processors, tensor processing units (TPUs), graphics processing units (GPUs), application specific integrated circuits, etc.). The one or more processors 1903 may be configured to execute computer readable instructions 1806 contained in the data storage device 1904 and / or other instructions as described herein.

[0176] The data storage device 1904 may include one or more non-transitory computer-readable storage media that can be read and / or accessed by at least one of the one or more processors 1903. The one or more computer-readable storage media may include volatile and / or non-volatile storage components, such as optical, magnetic, organic or other memory or disk storage devices, that may be fully or partially integrated with at least one of the one or more processors 1903. In some examples, the data storage device 1904 may be implemented using a single physical device (e.g., one optical, magnetic, organic or other memory or disk storage unit), while in other examples, the data storage device 1904 may be implemented using two or more physical devices.

[0177] Data storage 1904 may include computer-readable instructions 1806 and possibly additional data. In some examples, data storage 1904 may include storage required to perform at least a portion of the methods, scenarios, and techniques described herein and / or at least a portion of the functionality of the devices and networks described herein. In some examples, data storage 1904 may include storage for trained neural network models 1910 (e.g., models of trained neural networks such as vision and language models). Specifically, in these examples, computer-readable instructions 1806 may include instructions that, when executed by one or more processors 1903, enable computing device 1900 to provide some or all of the functionality of trained neural network model 1910.

[0178] In some examples, computing device 1900 may include one or more cameras 1912. Cameras 1912 may include one or more image capture devices, such as still cameras and / or video cameras, that are configured to capture light and record the captured light in one or more images; that is, camera 1912 may generate images of the captured light. The one or more images may be one or more still images and / or one or more images used in a video image. Camera 1912 may capture light and / or electromagnetic radiation emitted as visible light, infrared radiation, ultraviolet light, and / or as one or more other frequencies of light.

[0179] In some examples, computing device 1900 may include one or more sensors 1914. Sensors 1914 may be configured to measure conditions within computing device 1900 and / or conditions in the environment of computing device 1900 and provide data about these conditions. For example, sensors 1914 may include one or more of the following: (i) sensors for obtaining data about computing device 1900, such as, but not limited to, a thermometer for measuring the temperature of computing device 1900, a battery sensor for measuring the charge of one or more batteries of power system 1916, and / or other sensors for measuring conditions of computing device 1900; (ii) identification sensors for identifying other objects and / or devices, such as, but not limited to, radio frequency identification (RFID) readers, proximity sensors, one-dimensional barcode readers, two-dimensional barcode (e.g., quick response (QR) code) readers, and laser trackers, where identification sensors may be configured to read identifiers (such as RFID tags, barcodes, QR codes) and / or other devices and / or objects that are configured to be read and provide at least identification information; (iii) sensors for measuring the location of computing device 1900; (iv) an environmental sensor for obtaining data indicative of the environment of the computing device 1900, such as, but not limited to, an infrared sensor, an optical sensor, a light sensor, a biosensor, a capacitive sensor, a touch sensor, a temperature sensor, a wireless sensor, a radio sensor, a motion sensor, a microphone, a sound sensor, an ultrasonic sensor, and / or a smoke sensor; and / or (v) a force sensor for measuring one or more forces (e.g., inertial forces and / or gravity) acting on the computing device 1900, such as, but not limited to, one or more sensors measuring force, torque, ground force, friction in one or more dimensions, and / or a ZMP sensor that identifies a zero moment point (ZMP) and / or the location of the ZMP. Many other examples of sensors 1914 are also possible.

[0180] The power system 1916 may include one or more batteries 1918 and / or one or more external power interfaces 1920 for providing power to the computing device 1900. Each of the one or more batteries 1918 may act as a storage power source for the computing device 1900 when electrically coupled to the computing device 1900. The one or more batteries 1918 of the power system 1916 may be configured to be portable. Some or all of the one or more batteries 1918 may be easily removable from the computing device 1900. In other examples, some or all of the one or more batteries 1918 may be located inside the computing device 1900 and therefore may not be easily removable from the computing device 1900. Some or all of the one or more batteries 1918 may be rechargeable. For example, a rechargeable battery may be charged via a wired connection between the battery and another power supply device (such as by one or more power supply devices located outside the computing device 1900 and connected to the computing device 1900 via one or more external power interfaces). In other examples, some or all of the one or more batteries 1918 may be non-rechargeable batteries.

[0181] The one or more external power interfaces 1920 of the power system 1916 may include one or more wired power interfaces, such as USB cables and / or power cords, that implement a wired power connection to one or more power supply devices located outside the computing device 1900. The one or more external power interfaces 1920 may include one or more wireless power interfaces, such as Qi wireless chargers, that implement wireless power connections to one or more external power supply devices. Once a power connection to an external power source is established using the one or more external power interfaces 1920, the computing device 1900 can draw power from the external power source through the established power connection. In some examples, the power system 1916 may include associated sensors, such as battery sensors or other types of power sensors associated with one or more batteries.

[0182] Cloud-based servers

[0183] Fig. 20 A cloud-based server system is depicted according to an example embodiment. Fig. 20, the functionality and / or computing devices of open vocabulary object detection based on frozen visual and language models can be distributed among computing clusters 2009a, 2009b, 2009c. Computing cluster 2009a can include one or more computing devices 2000a, cluster storage array 2010a, and cluster router 2011a connected via a local cluster network 2012a. Similarly, computing cluster 2009b can include one or more computing devices 2000b, cluster storage array 2010b, and cluster router 2011b connected via a local cluster network 2012b. Likewise, computing cluster 2009c can include one or more computing devices 2000c, cluster storage array 2010c, and cluster router 2011c connected via a local cluster network 2012c.

[0184] In some embodiments, computing clusters 2009a, 2009b, 2009c may be a single computing device residing in a single computing center. In other embodiments, computing clusters 2009a, 2009b, 2009c may include multiple computing devices in a single computing center, or even multiple computing devices located in multiple computing centers located in different geographic locations. For example, Fig. 20 Each of the computing clusters 2009a, 2009b, 2009c is depicted as residing in a different physical location.

[0185] In some embodiments, the data and services at the computing clusters 2009a, 2009b, 2009c may be encoded as computer-readable information stored in a non-transitory tangible computer-readable medium (or computer-readable storage medium) and accessible by other computing devices. In some embodiments, the computing clusters 2009a, 2009b, 2009c may be stored on a single disk drive or other tangible storage medium, or may be implemented on multiple disk drives or other tangible storage media located at one or more different geographic locations.

[0186] In some embodiments, each of computing clusters 2009a, 2009b, and 2009c may have an equal number of computing devices, an equal number of cluster storage arrays, and an equal number of cluster routers. However, in other embodiments, each computing cluster may have a different number of computing devices, a different number of cluster storage arrays, and a different number of cluster routers. The number of computing devices, cluster storage arrays, and cluster routers in each computing cluster may depend on one or more computing tasks assigned to each computing cluster.

[0187] For example, in computing cluster 2009a, computing device 2000a can be configured to perform various computing tasks of a neural network (e.g., a detector head) and / or computing device based on a frozen vision and language model. In one embodiment, various functionalities of the neural network and / or computing device can be distributed among one or more of computing devices 2000a, 2000b, 2000c. The computing devices 2000b and 2000c in the respective computing clusters 2009b and 2009c can be configured similarly to the computing device 2000a in computing cluster 2009a. On the other hand, in some embodiments, computing devices 2000a, 2000b, and 2000c can be configured to perform different functions.

[0188] In some embodiments, computing tasks and stored data associated with the neural network and / or computing devices may be distributed across computing devices 2000a, 2000b, and 2000c based at least in part on processing requirements of the neural network and / or computing devices, processing capabilities of computing devices 2000a, 2000b, 2000c, latency of network links between computing devices in each computing cluster and between the computing clusters themselves, and / or other factors that may impact cost, speed, fault tolerance, resilience, efficiency, and / or other design goals of the overall system architecture.

[0189] The cluster storage array 2010a, 2010b, 2010c of the computing cluster 2009a, 2009b, 2009c may be a data storage array including a disk array controller configured to manage read and write access to a group of hard disk drives. The disk array controllers, alone or in conjunction with their respective computing devices, may also be configured to manage backup or redundant copies of data stored in the cluster storage array to protect against disk drive or other cluster storage array failures and / or network failures that prevent one or more computing devices from accessing one or more cluster storage arrays.

[0190] Similar to the way that the functionality of the conditional-based axial self-attention neural network and / or computing devices can be distributed across the computing devices 2000a, 2000b, 2000c of the computing clusters 2009a, 2009b, 2009c, the various active and / or backup portions of these components can be distributed across the cluster storage arrays 2010a, 2010b, 2010c. For example, some cluster storage arrays can be configured to store a first layer of the neural network and / or a portion of the data of the computing device, while other cluster storage arrays can store a second layer of the neural network and / or other portions of the data of the computing device. Also, for example, some cluster storage arrays can be configured to store data for the encoder of the neural network, while other cluster storage arrays can store data for the decoder of the neural network. In addition, some cluster storage arrays can be configured to store backup versions of data stored in other cluster storage arrays.

[0191] The cluster routers 2011a, 2011b, 2011c in the computing clusters 2009a, 2009b, 2009c may include networking devices configured to provide internal and external communications for the computing clusters. For example, the cluster router 2011a in the computing cluster 2009a may include one or more Internet switches and routing devices configured to (i) provide local area network communications between the computing device 2000a and the cluster storage array 2010a via a local cluster network 2012a, and (ii) provide wide area network communications between the computing cluster 2009a and the computing clusters 2009b and 2009c via a wide area network link 2013a to the network 1806. Cluster routers 2011b and 2011c may include similar network devices as cluster router 2011a, and cluster routers 2011b and 2011c may perform similar networking functions for computing clusters 2009b and 2009b as cluster router 2011a performs for computing cluster 2009a.

[0192] In some embodiments, the configuration of the cluster routers 2011a, 2011b, 2011c can be based at least in part on the data communication requirements of the computing devices and the cluster storage arrays, the data communication capabilities of the network devices in the cluster routers 2011a, 2011b, 2011c, the latency and throughput of the local cluster network 2012a, 2012b, 2012c, the latency, throughput and cost of the wide area network links 2013a, 2013b, 2013c, and / or other factors that may affect the cost, speed, fault tolerance, resilience, efficiency and / or other design criteria of the adjusted system architecture.

[0193] Example methods of operation

[0194] Fig.2121 is a flow diagram of a method 2100 according to an example embodiment. The method 2100 may be performed by a computing device, such as computing device 1900. The method 2100 may begin at block 2110, where the computing device receives a frozen vision and language model (VLM) pre-trained on a plurality of image-text pairs.

[0195] At box 2120, the computing device determines a detection region embedding indicating one or more regions of interest in the image for the image encoder pre-trained by the frozen VLM and the image embedding generated by the detector head.

[0196] At box 2130, the computing device generates text embeddings for the training object categories from the pre-trained text encoder of the frozen VLM.

[0197] At box 2140, the computing device trains the detector head to perform object detection on the training object category based on the frozen VLM, so that the detector head predicts objects from the target object vocabulary associated with the training object category based on the detection region embeddings and the text embeddings of the training object category.

[0198] At box 2150, the computing device provides a pre-trained frozen VLM and a trained detector head.

[0199] In some embodiments, prediction of an object involves determining, by a detector head, one or more detection scores for one or more regions of interest, wherein the one or more detection scores for the one or more regions of interest are indicative of the predicted object.

[0200] In some embodiments, prediction of objects involves training a detector head to predict one or more object detection boxes and associated masks corresponding to one or more regions of interest, and wherein one or more detection scores are associated with the one or more predicted object detection boxes.

[0201] In some embodiments, the training of the detector head may be based on one or more of a box region loss, a box classification loss, or a mask classification loss.

[0202] In some embodiments, the detector head comprises a first stage and a second stage, and wherein the determination of the detection region embedding is performed by the first stage, and wherein the determination of the text embedding is performed by the second stage.

[0203] In some embodiments, the detector head can be a neural network. In some embodiments, the detector head can be one of Mask R-CNN or Faster R-CNN.

[0204] In some embodiments, the detector head comprises a feature pyramid network.

[0205] In some embodiments, a pre-trained text encoder and a pre-trained image encoder of a frozen VLM may be jointly trained based on contrastive learning.

[0206] In some embodiments, the pre-trained image encoder includes (i) a feature extractor for generating an image representation of an image, and (ii) a feature pooling layer.

[0207] In some embodiments, the feature extractor comprises a ResNet-50 architecture.

[0208] In some embodiments, the feature pooling layer can be an attention layer of an image encoder.

[0209] Some embodiments are directed to maintaining an image normalization scheme for a pre-trained frozen VLM to achieve open vocabulary object detection.

[0210] Some embodiments involve receiving, by a computing device, a pre-trained frozen VLM and a trained detector head, the pre-trained frozen VLM having been trained on a set of image-text pairs of a first dataset, and the detector head having been trained on a first set of training object categories of the first dataset. Such embodiments involve testing, by the computing device, the detector head on a second dataset including a second set of inferred object categories, wherein the testing includes replacing the first set of training object categories of the first training dataset with the second set of inferred object categories of the second dataset.

[0211] Fig. 22 is a flow diagram of a method 2200 according to an example embodiment. The method 2200 may be performed by a computing device, such as computing device 1900. The method 2200 may begin at block 2210, where the computing device receives an input image.

[0212] At box 2220, the computing device applies a trained neural network for object detection, wherein the neural network includes a frozen vision and language model (VLM) pre-trained on multiple image-text pairs, and a trained detector head associated with the pre-trained frozen VLM and pre-trained on training object categories.

[0213] At box 2230, the computing device determines a detection region embedding indicating one or more regions of interest in the input image for the pre-trained image encoder of the frozen VLM and the image embedding generated by the detector head.

[0214] At box 2240, the computing device predicts, by the detector head, an object from a target object vocabulary associated with the training object category based on the detection region embeddings and the text embeddings of the training object category.

[0215] At box 2250, the computing device provides an object from the target object vocabulary to the input image.

[0216] In some embodiments, prediction of an object involves determining, by a detector head, one or more detection scores for one or more regions of interest, wherein the one or more detection scores for the one or more regions of interest are indicative of the predicted object.

[0217] In some embodiments, prediction of an object involves predicting one or more object detection boxes and associated masks corresponding to the one or more regions of interest, and wherein one or more detection scores are associated with the one or more predicted object detection boxes.

[0218] Some embodiments involve providing, by a computing device, a predicted object from a target object vocabulary associated with a training object class.

[0219] Some embodiments involve receiving, by a pre-trained text encoder, an inferred object category different from the training object category. Such embodiments involve augmenting, by a trained detector head, a text embedding of a training object category with an additional embedding of the inferred object category, and wherein prediction of the object comprises predicting the additional object from an augmented target object vocabulary associated with the training object category and the inferred object category based on the augmented text embedding and the detection region embedding.

[0220] In some embodiments, the prediction of the additional object involves determining, by the trained detector head, one or more enhanced detection scores for the one or more regions of interest, wherein the one or more enhanced detection scores for the one or more regions of interest are indicative of the predicted additional object.

[0221] In some embodiments, the determination of one or more open vocabulary detection scores involves determining a geometric mean of the one or more enhanced detection scores and the one or more VLM scores.

[0222] In some embodiments, the pre-trained image encoder includes a feature pooling layer trained to generate one or more VLM region embeddings. Such embodiments involve generating one or more VLM scores corresponding to one or more regions of interest by the feature pooling layer, wherein the one or more VLM scores are based on the one or more VLM region embeddings and the enhanced text embeddings. Such embodiments also involve determining one or more open vocabulary detection scores corresponding to the one or more regions of interest, wherein the one or more open vocabulary detection scores are based on the one or more enhanced detection scores and the one or more VLM scores.

[0223] In some embodiments, the feature pooling layer can be an attention layer of an image encoder.

[0224] In some embodiments, the detector head can be a neural network. In some embodiments, the detector head can be one of Mask R-CNN or Faster R-CNN.

[0225] In some embodiments, the detector head may have been trained to perform single-stage object detection.

[0226] In some embodiments, the detector head may have been trained to perform two-stage object detection. The determination of the detection region embedding may be performed by the first stage, and the determination of the text embedding may be performed by the second stage.

[0227] In some embodiments, the detector head may include a feature pyramid network.

[0228] In some embodiments, a pre-trained text encoder and a pre-trained image encoder of the frozen VLM may have been jointly trained based on contrastive learning.

[0229] The present disclosure is not limited in terms of the specific embodiments described in this application, which are intended to be illustrative of various aspects. As will be apparent to those skilled in the art, many modifications and variations can be made without departing from its spirit and scope. In addition to those methods and devices listed herein, functionally equivalent methods and devices within the scope of the present disclosure will be apparent to those skilled in the art from the foregoing description. Such modifications and variations are intended to fall within the scope of the appended claims.

[0230] The above detailed description describes various features and functions of the disclosed systems, devices, and methods with reference to the accompanying drawings. In the drawings, similar symbols generally identify similar components unless the context dictates otherwise. The illustrative embodiments described in the detailed description, drawings, and claims are not intended to be limiting. Other embodiments may be utilized and other changes may be made without departing from the spirit or scope of the subject matter presented herein. It will be readily understood that the aspects of the present disclosure as generally described herein and shown in the accompanying drawings may be arranged, substituted, combined, separated, and designed in a variety of different configurations, all of which are expressly contemplated herein.

[0231] With respect to any or all of the ladder diagrams, scenarios, and flow charts in the figures and discussed herein, each block and / or communication may represent information processing and / or information transmission according to an example embodiment. Alternative embodiments are included within the scope of these example embodiments. In these alternative embodiments, for example, the functions described as blocks, transmissions, communications, requests, responses, and / or messages may not be performed in the order shown or discussed, including substantially simultaneously or in reverse order, depending on the functionality involved. In addition, more or fewer blocks and / or functions may be used with any of the ladder diagrams, scenarios, and flow charts discussed herein. Figure 1 and these ladder diagrams, scenarios, and flow charts can be combined with each other in part or in whole.

[0232] The box representing information processing may correspond to a circuit that can be configured to perform a specific logical function of the method or technology described herein. Alternatively or in addition, the box representing information processing may correspond to a module, a segment, or a portion of program code (including related data). The program code may include one or more instructions that may be executed by a processor to implement a specific logical function or action in a method or technology. The program code and / or related data may be stored on any type of computer-readable medium, such as a storage device including a disk or hard drive or other storage medium.

[0233] Computer readable media may also include non-transitory computer readable media, such as non-transitory computer readable media for short-term storage of data, such as register memory, processor cache, and random access memory (RAM). Computer readable media may also include non-transitory computer readable media for long-term storage of program code and / or data, such as auxiliary or persistent long-term storage devices, for example, such as read-only memory (ROM), optical or magnetic disks, compact disk read-only memory (CD-ROM). Computer readable media may also be any other volatile or non-volatile storage system. Computer readable media may be considered, for example, as a computer readable storage medium or a tangible storage device.

[0234] In addition, the blocks representing one or more information transfers may correspond to information transfers between software and / or hardware modules in the same physical device. However, other information transfers may be between software modules and / or hardware modules in different physical devices.

[0235] As described in this paper, new capabilities for detecting previously unseen object categories (unseen by machine learning models) are described, particularly on challenging benchmark and transfer settings. The described models exploit information embedded in vision-language models that, in some cases, may reinforce flaws and biases in Internet data and / or propagate potentially harmful biases and / or stereotypes. Although the example models in this paper are trained for evaluation and / or benchmark purposes, such models typically require more rigorous probing for bias, fairness, etc. before use for other purposes.

[0236] While various aspects and embodiments have been disclosed herein, other aspects and embodiments will be apparent to those skilled in the art. The various aspects and embodiments disclosed herein are provided for purposes of illustration and are not intended to be limiting, with the true scope being indicated by the claims.

Claims

1. A computer-implemented method for training a detector head for object detection for training object categories based on a frozen vision and language model (VLM), comprising: Receiving, by a computing device, the frozen VLM pre-trained on a plurality of image-text pairs; for an image encoder pre-trained by the frozen VLM and an image embedding generated by the detector head, determining a detection region embedding indicative of one or more regions of interest in an image; Generating text embeddings of the training object categories by a pre-trained text encoder of the frozen VLM; predicting, by the detector head, an object from a target object vocabulary associated with the training object category based on the detection region embeddings and the text embeddings of the training object category; as well as A pre-trained frozen VLM and a trained detector head are provided by the computing device.

2. The computer-implemented method of claim 1 , wherein the prediction of the object comprises: One or more detection scores for the one or more regions of interest are determined by the detector head, wherein the one or more detection scores for the one or more regions of interest are indicative of a predicted object.

3. A computer-implemented method as described in claim 2, wherein the prediction of the object includes training the detector head to predict one or more object detection boxes and associated masks corresponding to the one or more regions of interest, and wherein the one or more detection scores are associated with one or more predicted object detection boxes. 4 . The computer-implemented method of claim 1 , wherein the training of the detector head is based on one or more of a box region loss, a box classification loss, or a mask classification loss.

5. The computer-implemented method of claim 1, wherein the detector head comprises a first stage and a second stage, and wherein the determination of the detection region embedding is performed by the first stage, and wherein the determination of the text embedding is performed by the second stage.

6. The computer implemented method of claim 1, wherein the detector head is a neural network.

7. The computer-implemented method of claim 6, wherein the detector head is one of Mask R-CNN or Faster R-CNN.

8. The computer-implemented method of claim 1, wherein the detector head further comprises a feature pyramid network.

9. The computer-implemented method of claim 1, wherein the pre-trained text encoder and the pre-trained image encoder of the frozen VLM are jointly trained based on contrastive learning.

10. The computer-implemented method of claim 1, wherein the pre-trained image encoder comprises (i) a feature extractor for generating an image representation of the image, and (ii) a feature pooling layer.

11. The computer-implemented method of claim 10, wherein the feature extractor comprises a ResNet-50 architecture.

12. The computer-implemented method of claim 10, wherein the feature pooling layer is an attention layer of the image encoder.

13. The computer-implemented method of claim 1, further comprising: The image normalization scheme of the pre-trained frozen VLM is maintained for open vocabulary object detection.

14. The computer-implemented method of claim 1, further comprising: receiving, by the computing device, the pre-trained frozen VLM and the trained detector head, the pre-trained frozen VLM having been trained on a set of image-text pairs of a first data set, and the detector head having been trained on a first set of training object categories of the first data set; as well as The detector head is tested by the computing device on a second dataset comprising a second set of inferred object categories, wherein the testing comprises replacing the first set of training object categories of the first training dataset with the second set of inferred object categories of the second dataset.

15. A computer-implemented method for applying a trained detector head to perform object detection on a training object class based on a frozen vision and language model (VLM), comprising: Receiving an input image by a computing device; applying a trained neural network for object detection, wherein the neural network comprises the frozen VLM pre-trained on a plurality of image-text pairs, and the trained detector head associated with the pre-trained frozen VLM and pre-trained on the training object categories; for an image encoder pre-trained by the frozen VLM and an image embedding generated by the detector head, determining a detection region embedding indicative of one or more regions of interest in the input image; predicting, by the detector head, an object from a target object vocabulary associated with the training object category based on the detection region embeddings and the text embeddings of the training object category; as well as The input image is provided with the object from the target object vocabulary by the computing device.

16. The computer-implemented method of claim 15, wherein the prediction of the object comprises: One or more detection scores for the one or more regions of interest are determined by the trained detector head, wherein the one or more detection scores for the one or more regions of interest are indicative of a predicted object.

17. The computer-implemented method of claim 16, wherein the prediction of the object comprises predicting one or more object detection boxes and associated masks corresponding to the one or more regions of interest, and wherein the one or more detection scores are associated with the one or more predicted object detection boxes.

18. The computer-implemented method of claim 15, further comprising: The predicted object from the target object vocabulary associated with the training object class is provided by the computing device.

19. The computer-implemented method of claim 15, further comprising: Receiving, by a pre-trained text encoder, an inference object category different from the training object category; augmenting, by the trained detector head, the text embeddings of the training object categories with additional embeddings of the inferred object categories, and Wherein the prediction of the object comprises predicting additional objects from an enhanced target object vocabulary associated with the training object category and the inference object category based on the enhanced text embedding and the detection region embedding.

20. The computer-implemented method of claim 19, wherein the prediction of the additional object comprises: One or more enhanced detection scores for the one or more regions of interest are determined by the trained detector head, wherein the one or more enhanced detection scores for the one or more regions of interest are indicative of the predicted additional objects.

21. The computer-implemented method of claim 20, wherein the pre-trained image encoder comprises a feature pooling layer trained to generate one or more VLM region embeddings, and the method further comprises: generating, by the feature pooling layer, one or more VLM scores corresponding to the one or more regions of interest, wherein the one or more VLM scores are based on the one or more VLM region embeddings and the enhanced text embedding; and One or more open vocabulary detection scores corresponding to the one or more regions of interest are determined, wherein the one or more open vocabulary detection scores are based on the one or more enhanced detection scores and the one or more VLM scores.

22. The computer-implemented method of claim 21, wherein the determining of the one or more open vocabulary detection scores comprises determining a geometric mean of the one or more enhanced detection scores and the one or more VLM scores.

23. The computer-implemented method of claim 21, wherein the feature pooling layer is an attention layer of the pre-trained image encoder.

24. The computer implemented method of claim 15, wherein the detector head is a neural network.

25. The computer-implemented method of claim 24, wherein the detector head is one of Mask R-CNN or Faster R-CNN.

26. The computer implemented method of claim 24, the detector head having been trained to perform single-stage object detection.

27. The computer-implemented method of claim 24, wherein the detector head has been trained to perform two-stage object detection, wherein the determination of the detection region embedding is performed by a first stage, and wherein the determination of the text embedding is performed by a second stage.

28. The computer-implemented method of claim 15, wherein the detector head further comprises a feature pyramid network.

29. The computer-implemented method of claim 1, wherein the pre-trained text encoder and the pre-trained image encoder of the frozen VLM have been jointly trained based on contrastive learning.

30. A computing device comprising: one or more processors; as well as A data storage device having computer executable instructions stored thereon, which when executed by the one or more processors cause the computing device to perform functions including the computer-implemented method of any one of claims 1 to 29.

31. The computing device of claim 30, wherein the computing device is a mobile phone.

32. A computer program comprising instructions which, when executed by a computer, cause the computer to perform the steps of the method according to any one of claims 1 to 29.

33. An article comprising: One or more non-transitory computer-readable media having computer-readable instructions stored thereon, which when executed by one or more processors of a computing device cause the computing device to perform functions including the computer-implemented method of any one of claims 1 to 29.

34. A system comprising: Means for performing the computer implemented method of any one of claims 1 to 29.

Images