Object detection using visual language models via latent feature adaptation with synthetic data

Abstract

Systems and techniques for adapting pre-trained machine learning models are described herein. For example, a method can include encoding a training image as a first feature vector, the training image including a first object located at a first position; generating a second feature vector based on a set of sinusoidal functions using a set of weights; combining the first feature vector with the second feature vector to generate a combined feature vector; processing the combined feature vector using a visual language model to obtain a second position of the first object; and adjusting the set of weights based on a comparison between the first position and the second position.

Description

Technical Field

[0001] This disclosure relates in general to object detection. For example, aspects of this disclosure relate to systems and techniques for performing object detection by adapting a visual language model to utilize latent features of synthetic data. Background Technology

[0002] A Visual Language Model (VLM) is an artificial intelligence / machine learning (AI / ML) model designed to process visual content (such as images, videos, etc.) to learn to recognize and classify visual elements (such as objects, scenes, styles, etc.) and generate text based on the visual content. For example, when combined with a Large Language Model (LLM), a VLM can be able to classify visual elements in visual content or answer questions about the visual content.

[0003] LLMs are examples of ML models trained to perform natural language processing tasks such as generating text, predicting text, translating text, etc. In some cases, LLMs can be implemented using neural networks (NNs) and / or transformer models. A transformer model is a type of ML model that includes an encoder and a decoder and can be used to tokenize input, learn relationships between tokens, and then use the tokens to generate predictions. As an example, a VLM might be able to generate text output indicating what objects are in an image, or generate an image based on text input.

[0004] However, while a Visual Model (VLM) can indicate that an image contains an object, it often struggles to pinpoint the object's location within the image. For example, a VLM might be able to indicate that an image includes a dog, but it might not be able to localize the dog to indicate that it is in the lower left corner (e.g., within the box between pixels (X1, Y1) and (X2, Y2)). This localization task can be part of a computer vision task related to object detection. In some cases, techniques used to allow existing VLM models (e.g., pre-trained VLM models that have not been retrained) to perform object detection can be useful. Summary of the Invention

[0005] The following is a simplified summary of the invention relating to one or more aspects disclosed herein. Therefore, this summary should not be considered an exhaustive overview relating to all conceived aspects, nor should it be considered to identify key or decisive elements relating to all conceived aspects or to depict the scope associated with any particular aspect. Accordingly, the following summary presents certain concepts in a simplified form relating to one or more aspects of the mechanisms disclosed herein, preceding the detailed description that follows.

[0006] This paper describes systems and techniques for performing object detection by adapting visual language models to utilize latent features of synthetic data.

[0007] In one exemplary example, an apparatus for adapting a pre-trained machine learning model is provided. The apparatus includes: one or more memories; and one or more processors coupled to the memories and configured to: encode a training image into a first feature vector, the training image including a first object located at a first position; generate a second feature vector using a set of weights based on a set of sinusoidal functions; combine the first feature vector with the second feature vector to generate a combined feature vector; process the combined feature vector using a visual language model to obtain a second position of the first object; and adjust the set of weights based on a comparison between the first position and the second position.

[0008] As another example, a method for adapting a pre-trained machine learning model is provided. The method includes: encoding a training image into a first feature vector, the training image including a first object located at a first position; generating a second feature vector using a set of weights based on a set of sinusoidal functions; combining the first feature vector with the second feature vector to generate a combined feature vector; processing the combined feature vector using a visual language model to obtain a second position of the first object; and adjusting the set of weights based on a comparison between the first position and the second position.

[0009] In another example, a non-transitory computer-readable medium is provided having instructions stored thereon. These instructions, when executed by one or more processors, cause the processors to: encode a training image into a first feature vector, the training image including a first object located at a first position; generate a second feature vector using a set of weights based on a set of sinusoidal functions; combine the first feature vector with the second feature vector to generate a combined feature vector; process the combined feature vector using a visual language model to obtain a second position of the first object; and adjust the set of weights based on a comparison between the first position and the second position.

[0010] As another example, an apparatus for adapting a pre-trained machine learning model is provided. The apparatus includes: components for encoding a training image into a first feature vector, the training image including a first object located at a first position; components for generating a second feature vector using a set of weights based on a set of sinusoidal functions; components for combining the first feature vector and the second feature vector to generate a combined feature vector; components for processing the combined feature vector using a visual language model to obtain a second position of the first object; and components for adjusting the set of weights based on a comparison between the first position and the second position.

[0011] In another example, an apparatus for image processing is provided. The apparatus includes: one or more memories; and one or more processors coupled to the memories and configured to: process an image and a first text cue using a visual language model to obtain a list of objects in the image; process the image and a second text cue using an adapted visual language model, the second text cue including a first object from the object list; and receive a set of coordinates of the first object in the image.

[0012] As another example, a method for image processing includes: processing an image and a first text cue using a visual language model to obtain a list of objects in the image; processing the image and a second text cue using an adapted visual language model, the second text cue including a first object from the list of objects; and receiving a set of coordinates of the first object in the image.

[0013] In another example, a non-transitory computer-readable medium is provided on which instructions are stored. These instructions, when executed by one or more processors, cause the processors to: process an image and a first text prompt using a visual language model to obtain a list of objects in the image; process the image and a second text prompt using an adapted visual language model, the second text prompt including a first object from the list of objects; and receive a set of coordinates of the first object in the image.

[0014] As another example, an apparatus for image processing is provided. The apparatus includes: components for processing an image and a first text prompt using a visual language model to obtain a list of objects in the image; components for processing the image and a second text prompt using an adapted visual language model, the second text prompt including a first object from the object list; and components for receiving a set of coordinates of the first object in the image.

[0015] In some aspects, one or more of the devices described herein include mobile devices (e.g., mobile phones or so-called "smartphones," tablet computers, or other types of mobile devices), wearable devices, extended reality devices (e.g., virtual reality (VR) devices, augmented reality (AR) devices, or mixed reality (MR) devices), personal computers, laptop computers, video servers, television sets (e.g., network-connected television sets), vehicles (or computing devices within vehicles), or other devices. In some aspects, the device includes at least one camera for capturing one or more images or video frames. For example, the device may include one or more cameras (e.g., an RGB camera) for capturing one or more images and / or one or more videos including video frames. In some aspects, the device may include a display for displaying one or more images, videos, notifications, or other displayable data. In some aspects, the device may include a transmitter configured to transmit one or more video frames and / or syntax data to at least one device via a transmission medium. In some aspects, the processor includes a neural processing unit (NPU), a central processing unit (CPU), a graphics processing unit (GPU), or other processing devices or components.

[0016] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in isolation to define the scope of the claimed subject matter. This subject matter should be understood with reference to the appropriate portions of the entire specification, any or all drawings, and each claim.

[0017] The foregoing, as well as other features and embodiments, will become more apparent from the following description, claims and drawings. Attached Figure Description

[0018] The exemplary embodiments of this application are described in detail below with reference to the following figures: Figure 1 Example implementations of a System-on-a-Chip (SOC) based on some examples are illustrated; Figure 2 This is a block diagram illustrating techniques for adapting a VLM to perform object detection using latent feature adaptation according to various aspects of this disclosure; Figure 3 This is a block diagram illustrating the generation of synthetic data according to various aspects of this disclosure; Figure 4 This is a block diagram illustrating the use of an adapted VLM for object detection according to various aspects of this disclosure; Figure 5 This is a flowchart illustrating a process for adapting a pre-trained ML model according to various aspects of this disclosure; Figure 6This is a flowchart illustrating a process for image processing according to various aspects of this disclosure; Figure 7 Two sets of images are provided, based on several aspects, illustrating the forward diffusion process (which is fixed) and the reverse diffusion process (which is learned) of the diffusion model. Figure 8 This is a diagram illustrating how a diffusion model can be used to distribute diffused data from initial data to noise in the forward diffusion direction, based on several aspects. Figure 9 This is a diagram illustrating the U-Net architecture used for diffusion models based on some aspects; Figure 10 This is a block diagram illustrating examples of deep learning neural networks that can be used to implement a perception module and / or one or more verification modules, based on some aspects. Figure 11 This is a block diagram illustrating examples of convolutional neural networks (CNNs) according to various aspects of this disclosure; and Figure 12 Example computing device architectures are illustrated, showing example computing devices that can implement the various technologies described herein. Detailed Implementation

[0019] Certain aspects and embodiments of this disclosure are provided below. Some of these aspects and embodiments may be applied independently, and some may be combined, as will be apparent to those skilled in the art. Specific details are set forth in the following description for purposes of explanation in order to provide a thorough understanding of the various embodiments of this application. However, it will be apparent, however, that the various embodiments may be practiced without these specific details. The accompanying drawings and descriptions are not intended to be limiting.

[0020] The following description provides only exemplary embodiments and is not intended to limit the scope, applicability, or configuration of this disclosure. Rather, the following description of the exemplary embodiments will provide those skilled in the art with enabling descriptions for implementing the exemplary embodiments. It should be understood that various changes may be made to the function and arrangement of the elements without departing from the spirit and scope of this application as set forth in the appended claims.

[0021] In some cases, a pre-trained VLM with frozen weights (e.g., weights that were not trained during subsequent training processes) can be adapted to perform tasks that the original VLM might not be able to perform. In some cases, the general VLM may be relatively large, and a significant amount of computational resources can be used to train it. Since training a VLM from scratch can be expensive in terms of both computational resources and time, it can be useful to leverage an existing pre-trained VLM model with frozen weights to perform localization tasks for object detection.

[0022] This document describes systems, apparatuses, electronic devices, methods (also referred to as processes), and computer-readable media (collectively, “Systems and Technologies”) for adapting visual language models to object detection by leveraging latent features from synthetic data. For example, using a pre-trained ML model (such as a VLM model), training images including objects can be encoded as feature vectors. Another vector can be generated based on a set of weights and a set of sinusoidal functions. The weight set can be a trainable part of a lightweight CNN or other ML model. The vector and the other vector can be combined and submitted to the VLM along with a training cue. The training cue can request the location of objects in the image. The VLM can generate a result, which can be compared to the location of objects in the image. The weight set can be adjusted using the difference between the result and the location of objects in the image.

[0023] Various aspects of this disclosure will be described with reference to the figures.

[0024] Figure 1 An example implementation of a System-on-Chip (SOC) 100 is illustrated, which may include a Central Processing Unit (CPU) 102 or a multi-core CPU configured to perform one or more of the functions described herein. Parameters or variables (e.g., neural signals and synaptic weights), system parameters associated with computing devices (e.g., a weighted neural network), latency, frequency bin information, task information, and other information may be stored in a memory block associated with a Neural Processing Unit (NPU) 108, a memory block associated with the CPU 102, a memory block associated with a Graphics Processing Unit (GPU) 104, a memory block associated with a Digital Signal Processor (DSP) 106, a memory block 118, and / or may be distributed across multiple blocks. Instructions executed at the CPU 102 may be loaded from the program memory associated with the CPU 102 or from memory block 118.

[0025] The SOC 100 may also include additional processing blocks tailored for specific functions, such as a GPU 104, a DSP 106, a connectivity block 110 (which may include fifth-generation (5G) connectivity, fourth-generation LTE (4G) connectivity, Wi-Fi connectivity, USB connectivity, Bluetooth connectivity, etc.), and a multimedia processor 112 capable of, for example, detecting and recognizing gestures. In one implementation, the NPU is implemented within the CPU 102, DSP 106, and / or GPU 104. The SOC 100 may also include a sensor processor 114, an image signal processor (ISP) 116, and / or a navigation module 120, which may include a global positioning system.

[0026] SOC 100 may be based on the ARM instruction set. SOC 100 and / or its components may be configured to perform segmentation mask extrapolation. For example, CPU 102, DSP 106 and / or GPU 104 may be configured to perform object detection by adapting a visual language model using latent features of synthetic data.

[0027] In some cases, a VLM can accept visual data (such as images, videos, etc.) as well as text data to output text. For example, a VLM can accept visual data as input, such as images including dogs. I And text input asking whether a cat exists in an image. T Visual data (e.g., images) I (can be generated by a visual encoder) V Processing to generate feature vectors ,in N p Indicates the number of blocks, and D v This represents the channel dimension. Similarly, textual information T can be tokenized and embedded to produce text embeddings. ,in M Indicates the number of text tags, and D t Indicates the embedding dimension. Visual features. x v In relation to text features x t Processed together via a converged network F To be derived by language decoder D Generate response text indicating that there is no cat in the image. .

[0028] In some cases, a generalized VLM can be relatively large, and significant computational resources can be used to train it. Since training a VLM from scratch can be expensive in terms of both computational resources and time, it can be useful to leverage an existing pre-trained VLM model with frozen weights to perform a localization task for object detection. For example, the learned parameters can be used to adapt features from a frozen pre-trained VLM to perform a localization task for object detection.

[0029] In some cases, a pre-trained Virtual Model (VLM) (e.g., a VLM with frozen weights) can be adapted to perform tasks that the original VLM might not be able to perform. For example, a VLM that has been pre-trained on general data can support Low-Rank Adaptation (LoRA). In LoRA, a pre-trained frozen model (e.g., an ML model with frozen weights) can include trainable matrices in the layers of an ML model that allow the ML model to adapt to different tasks. In some cases, these trainable matrices can be trained to support localization tasks.

[0030] Figure 2 This is a block diagram illustrating a technique 200 for adapting a VLM to perform object detection using latent feature adaptation, according to various aspects of this disclosure. Figure 2 As shown, the pre-trained VLM 202 may also include a visual encoder 204 for LoRA. And a perceptron / resampler 206. Examples of VLMs may include Flamingo, CLIP, VisualBert, etc. VLM 202 may include VLM layers 1 220A to N 220N and trainable matrices 1 218A to N 218N. VLM 202 can be pre-trained for general tasks, such as generating text descriptions of whether an object is in an image, and VLM 202 can implement LoRA to allow VLM 202 to be adapted to perform other tasks.

[0031] The perceptron / resampler 206 can recompile the received feature vectors to a size and / or format compatible with the VLM 202, thus adapting the VLM 202 to a specific task. Visual encoder 204 The received image 208 can be used to encode a labeled feature vector 228 (e.g., a first feature vector). The labeled feature vector 228 from the visual encoder 204 can be combined with a spatial feature vector 230 (e.g., a second feature vector) from the localization insertion engine (PIE) 210.

[0032] PIE 210 can be a multilayer perceptron (MLP) trained to learn input-agnostic spatial feature vectors 230, which can be used with a visual encoder 204. The feature vectors 228 of the labeled output are combined. To infuse spatial awareness into the PIE 210, fixed-location embeddings (e.g., sinusoidal embeddings 212) can be used. In some cases, the sinusoidal embedding 212 can be based on the sinusoidal and cosine functions of the location. For example, the sinusoidal embedding 212 may have a dimension d And it uses a sinusoidal function: Equation 1: , Equation 2: , in i represents the index for positioning, and k represents the index within the embedded dimension, where d model As the dimension of the embedding space. k The range extends from 1 to d model The sinusoidal embedding 212 is used to indicate to the PIE 210 where in the feature space (e.g., the feature vectors of the labeled images from the visual encoder 204 PIE 210) the spatial feature vector 230 is generated. The sinusoidal embedding 212 can help limit the size of the PIE 210 when processing higher-dimensional vectors (such as the feature vectors 228 of the labeled images from the visual encoder 204 (e.g., greater than about 10 dimensions)). The PIE 210 can then learn the output of the sinusoidal embedding 212 and can tune the spatial feature vector 230 (e.g., a spatial sinusoidal vector) so that the VLM 202 can understand to output a description of the object. obj> The text of the bounding box (such as object 222).

[0033] In some cases, the spatial feature vector 230 generated by PIE 210 can be generated by a learnable shallow feedforward neural network. ψ To refine, thus obtaining v P =ψ ( S ), where the output dimension is the same as the output dimension from the visual encoder. Matching. Then, the learned spatial feature vector 230 can be added to the output from the visual encoder 204. ( I This results in a rich representation of visual features. A feedforward neural network can be trained during the learning phase based on a set of weights that can be adjusted as part of the training PIE 210. ψ The output of the PIE 210 is independent of the received image 208 and is deterministic.

[0034] In some cases, ψ PIE 210 parameters θ The text output 214 generated by the language decoder can be optimized. In some cases, the text output 214 may correspond to the text output of the coordinates of the bounding box around the predicted position of the located object. This process does not require an additional head or projection layer, thus maintaining the simplicity of the VLM and native natural language output. For example, the PIE 210 can utilize text prompts... tp ∈ T (such as going to VLM 202 "< obj The input sequence 216, consisting of ">located in the image"), is used for training, and the task of VLM 202 can be to utilize a given object name. <obj>The bounding box coordinates (such as a cup) complete the sequence. For a given object name existing within the image, < obj VLM can predict templates based on image features and initial text prompts. t r ∈ T The bounding box coordinate sequence in, such as [ x min , y min , x max , y max ].

[0035] As part of the training, the predicted bounding box coordinate sequence can be compared with the expected location of the bounding box based on the loss, and this loss can be used to adjust the feedforward neural network of PIE 210. ψ The weights. In some cases, negative log-likelihood loss can be used for the predicted labels. Negative log-likelihood can be...

[0036] in y t Corresponding to positioning in the text t The target marker at the location, T It is the total number of tags to be predicted, and v This is for locating and enhancing feature vectors. Here, p θ The model assigns the location t The probability of correctly labeling a given location, relative to previous labels. y <t The learning objective is conditioned on visual features and textual cues. This objective helps enable the pre-trained VLM to be used for localization without relying on specialized components such as region proposal networks. In some cases, VLM 202 can be trained for localization using synthetic data, and the received image 208 can be a synthetic image used for training. In some cases, synthetic images can be generated by using a synthetic data generator 226 to insert object 222 into other images 224.

[0037] Figure 3 This is a block diagram illustrating the generation of 300 from synthetic data based on various aspects of this disclosure. For example... Figure 3 As shown, object 302 can be generated, for example, using a text-to-image generation model (such as stable diffusion) based on a list of objects (such as a list of Large Vocabulary Instance Segmentation (LVIS) datasets for object categories). Object cues (e.g., object categories, such as "suitcase" or "cup") can be obtained from the list of LVIS datasets for object categories and submitted to the text-to-image generation model to generate an image of object 302.

[0038] In some cases, inappropriate objects can be filtered out (e.g., objects generated that do not match the object cues). For example, images can be submitted to a Contrastive Language Image Pre-training (CLIP) ML model. The CLIP ML model takes an image (such as an image of an object) and text (such as an object cue) and outputs a score indicating the relevance of the text to the object cues. If the score does not reach a threshold, the object image can be discarded. If the score reaches the threshold, the object image can be used for training.

[0039] At box 306, objects 302 can be randomly inserted based on a grid superimposed on the background image 304 to produce image 312. In some cases, the background image 304 can be a randomly picked image at a specific location. Additional objects 308 can also be inserted at box 310 to generate a composite image 314. Therefore, the composition function... C The composite image 314 can be returned by overlaying the object onto random pick locations in the background image 304, while taking into account the aspect ratio of the object. r Minimum size s min and maximum size x max Quantity of objects a and maximum overlap relative to already inserted objects o max It also returns text containing location information. Given a background image 304. I b ∈ I The composition function generates

[0040] The text prompt t ∈ T Including by C Randomly selected object positions and overlay images I p ∈ I This process produces results that can be used for training. Figure 2 The self-generated supervisory signal of PIE 210.

[0041] In some cases, machine learning networks can be trained using online training, offline training, and / or various combinations of online and offline training (e.g., such as...). Figure 2 The training of PIE 210, and various other machine learning networks. In some cases, online processing can refer to the processing of input data (e.g., such as received images 208, etc.) during its operation. Figure 2 The input sequence 216, etc., can be used, for example, as a time period for performing bounding box prediction processing implemented by the systems and techniques described herein. In some examples, offline may refer to an idle time period or a time period during which no input data is processed. Additionally, offline may be based on one or more time conditions (e.g., after a certain amount of time has elapsed, such as a day, a week, a month, etc.) and / or may be based on various other conditions, such as network and / or server availability, and various other conditions. In some aspects, offline training of a machine learning model (e.g., a neural network model) may be performed by a first device (e.g., a server device) to generate a pre-trained model, and a second device may receive the trained model from the second device. In some cases, a second device (e.g., a mobile device, XR device, vehicle or system / component of a vehicle, or other device) may perform online (or on-device) training of the pre-trained model to further adapt or tune the parameters of the model.

[0042] Figure 4 This is a block diagram illustrating the use of an adapted VLM for object detection according to various aspects of this disclosure. (e.g.) Figure 4 As shown, input image 402 can be input to VLM 404 along with text input 406, causing VLM 404 to generate a list 408 of objects that can be found in input image 402. VLM 404 can be an unadapted version of VLM 404. The output list 408 of objects can be used to populate text input 410 for an adapted version of the VLM (e.g., an adapted VLM 412). For example, one or more objects in the list 408 can be selected and inserted into text input 410. In some cases, the adapted VLM 412 can be the same as VLM 404, and one or more parameters, settings, input values, etc., can be used to indicate that an adapted version of VLM 404 can be used. Text input 410 can be input to the adapted VLM 412 to obtain bounding box position information 414 for one or more objects indicated in text input 410.

[0043] Figure 5 This is a flowchart illustrating a process 500 for adapting a pre-trained ML model according to various aspects of this disclosure. Process 500 may be performed by a computing device (or apparatus) (e.g., Figure 1 SOC 100 Figure 12 The computing device architecture 1200) or components of the computing device (e.g., chipset, codec, ... Figure 1 CPU 102, GPU 104, DSP 106, NPU 108, Figure 12 The process 500 can be executed by a processor 1210, etc. The computing device can be a mobile device (e.g., a mobile phone), a network-connected wearable device (such as a watch), an extended reality (XR) device (such as a virtual reality (VR) device or an augmented reality (AR) device), a vehicle or a component or system of a vehicle, or other types of computing devices. The operation of process 500 can be implemented as a software component that executes and runs on one or more processors.

[0044] At box 502, the computing device (or a component thereof) can (e.g., via...) Figure 2 The visual encoder 204 encodes the training image into a first feature vector, the training image including a first object (e.g., a suitcase) located at a first position. In some cases, the computing device (or a component thereof) can generate the training image. In some examples, the computing device (or a component thereof) can generate the training image by inserting the first object into a background image. In some cases, the computing device (or a component thereof) can generate the training image by generating an image of the first object. In some cases, the computing device (or a component thereof) can generate an image of the first object by submitting an object cue to the text-to-image generation model.

[0045] At box 504, the computing device (or a component thereof) may use (e.g., PIE 210) a weight set based on a set of sinusoidal functions (e.g., Equations 1 and 2) to (e.g., by...) Figure 2 The PIE 210 generates the second eigenvector. In some cases, the set of sinusoidal functions includes both cosine and sine functions.

[0046] At box 506, the computing device (or a component thereof) can combine the first feature vector with the second feature vector to generate a combined feature vector. In some cases, the combined feature vector can be (e.g., by...) Figure 2 The perceptron / resampler 206 is recomputed as a vector of a different size.

[0047] At box 508, the computing device (or a component thereof) may use a visual language model (e.g., Figure 2 The VLM 202) processes the combined feature vectors to obtain the second position of the first object (e.g., Figure 2 Text output 214, Figure 4 The bounding box location information 414). In some cases, the second location includes the text coordinates of the bounding box. In some cases, a visual language model is used to process the combined feature vectors and text cues (e.g., Figure 2 The input sequence 216 Figure 4 The text input (410) is used to obtain the second location of the first object. In some examples, the text prompt includes a hint about the location of the first object. In some cases, the weights of the visual language model are frozen.

[0048] At box 510, the computing device (or a component thereof) may adjust the weight set based on a comparison between the first position and the second position.

[0049] Figure 6 This is a flowchart illustrating a process 600 for image processing according to various aspects of the present disclosure. Process 600 may be performed by a computing device (or apparatus) (e.g., Figure 1 SOC 100 Figure 12 The computing device architecture 1200) or components of the computing device (e.g., chipset, codec, ... Figure 1 CPU 102, GPU 104, DSP 106, NPU 108, Figure 12 The process 600 is executed by a processor 1210, etc. The computing device can be a mobile device (e.g., a mobile phone), a network-connected wearable device (such as a watch), an extended reality (XR) device (such as a virtual reality (VR) device or an augmented reality (AR) device), a vehicle or a component or system of a vehicle, or other types of computing devices. The operation of process 600 can be implemented as a software component that executes and runs on one or more processors.

[0050] At box 602, the computing device (or a component thereof) can process the image (e.g., using a visual language model, such as VLM 404). Figure 4 The input image 402) and the first text prompt (e.g., Figure 4 The text input (406) is used to obtain a list of objects in the image. In some cases, the first text prompt includes a list of objects in the image. In some examples, the weights of the visual language model are frozen.

[0051] At box 604, the computing device (or a component thereof) can use an adapted visual language model (e.g., Figure 4 The VLM 412 adaptation) processes images and second text prompts (e.g., Figure 4 The text input (410) includes a second text prompt, which includes a first object from the object list (e.g., a suitcase). In some cases, the second text prompt includes a hint about the location of the first object. In some examples, the visual language model and the adapted visual language model are the same machine language model.

[0052] At box 606, the computing device (or a component thereof) may receive a set of coordinates of a first object in the image (e.g., Figure 4 (Bounding box location information 414).

[0053] In some examples, the techniques or processes described herein may be performed by a computing device or apparatus and / or any other computing device. In some cases, the computing device or apparatus may include a processor, microprocessor, microcomputer, or other components of a device configured to perform the steps of the processes described herein. In some examples, the computing device or apparatus may include a camera configured to capture video data (e.g., video sequences) including video frames. For example, the computing device may include a camera device that may or may not include a video codec. As another example, the computing device may include a mobile device with a camera (e.g., camera devices such as digital cameras, IP cameras, etc., mobile phones or tablet computers including cameras, or other types of devices with cameras). In some cases, the computing device may include a display for displaying images. In some examples, the camera or other capturing device that captures video data is separate from the computing device, in which case the computing device receives the captured video data. The computing device may also include a network interface, transceiver, and / or transmitter configured to communicate video data. The network interface, transceiver, and / or transmitter may be configured to communicate Internet Protocol (IP) based data or other network data.

[0054] The processes described herein can be implemented in hardware, computer instructions, or any combination thereof. In the context of computer instructions, each operation represents a computer-executable instruction stored on one or more computer-readable storage media that, when executed by one or more processors, performs the described operation. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc., that perform a particular function or implement a particular data type. The order in which the operations are described is not intended to be construed as limiting, and any number of the described operations can be combined in any order and / or in parallel to implement the process.

[0055] Additionally, the processes described herein can be configured to execute under the control of one or more computer systems using executable instructions, and can be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that executes jointly on one or more processors, implemented in hardware, or a combination of the foregoing. As noted above, the code can be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program comprising multiple instructions executable by one or more processors. The computer-readable or machine-readable storage medium can be non-transitory.

[0056] Machine learning (ML) can be considered a subset of artificial intelligence (AI). ML systems can include algorithms and statistical models that computer systems can use to perform various tasks through pattern-dependent inference and speculation, without explicit instructions. An example of an ML system is a neural network (also known as an artificial neural network), which can include groups of interconnected artificial neurons (e.g., neuron models). Neural networks can be used in a variety of applications and / or devices, such as image and / or video decoding, image analysis and / or computer vision applications, Internet Protocol (IP) cameras, Internet of Things (IoT) devices, autonomous vehicles, service robots, and more.

[0057] Individual nodes in a neural network mimic biological neurons by taking input data and performing simple operations on that data. The results of these simple operations on the input data are selectively passed to other neurons. Weights are associated with each vector and node in the network, and these values ​​constrain how the input data relates to the output data. For example, the input data of each node can be multiplied by its corresponding weight value, and the products can be summed. The sum of the products can be adjusted with optional biases, and activation functions can be applied to the results to produce the node's output signal or "output activation" (sometimes called a feature map or activation map). The weights can initially be determined by an iterative stream of training data through the network (e.g., weights are established during training phases where the network learns how to identify a particular category based on the characteristics of its typical input data).

[0058] There are different types of neural networks, such as Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), Generative Adversarial Networks (GANs), Multilayer Perceptron (MLP) neural networks, Transformer Neural Networks, and Diffusion-based Neural Networks. For example, a Convolutional Neural Network (CNN) is a feedforward artificial neural network. A CNN may comprise a collection of artificial neurons, each possessing a receptive field (e.g., a localized region of the input space) and collectively tiling the input space. RNNs work on the principle of storing the layer's output and feeding that output back to the input to help predict the layer's outcome. A GAN is a generative neural network that learns patterns in the input data so that the neural network model can generate new synthetic outputs, which may reasonably come from the original dataset. A GAN may comprise two neural networks operating together: a generative neural network that generates the synthetic output and a discriminative neural network that evaluates the authenticity of the output. In an MLP neural network, data is fed into the input layer, and one or more hidden layers provide an abstraction level to the data. The output layer can then be predicted based on this abstract data.

[0059] Deep learning (DL) is an example of machine learning techniques and can be considered a subset of ML. Many DL methods are based on neural networks, such as RNNs or CNNs, and utilize multiple layers. Using multiple layers in a deep neural network allows for the progressive extraction of higher-level features from a given raw data input. For example, the output of the first layer of artificial neurons becomes the input of the second layer, the output of the second layer becomes the input of the third layer, and so on. The layers located between the input and output of the entire deep neural network are often called hidden layers. Hidden layers learn (e.g., are trained) by transforming intermediate inputs from previous layers into slightly more abstract and complex representations that can be provided to subsequent layers until the final or desired representation is obtained as the final output of the deep neural network.

[0060] As noted above, neural networks are examples of machine learning systems and can include an input layer, one or more hidden layers, and an output layer. Data is provided from input nodes in the input layer, processed by hidden nodes in one or more hidden layers, and output is produced by output nodes in the output layer. Deep learning networks typically include multiple hidden layers. Each layer of a neural network can include a feature map or activation map, which can include artificial neurons (or nodes). Feature maps can include filters, kernels, etc. Nodes can include one or more weights used to indicate the importance of nodes in one or more layers. In some cases, deep learning networks may have a series of many hidden layers, where earlier layers are used to determine simple and low-level properties of the input, and later layers build a hierarchy of more complex and abstract properties.

[0061] Deep learning architectures can learn hierarchical structures of features. For example, if presented with visual data, the first layer can learn to recognize relatively simple features in the input stream, such as edges. In another example, if presented with auditory data, the first layer can learn to recognize spectral power at specific frequencies. The second layer, taking the output of the first layer as input, can learn to recognize combinations of features, such as simple shapes in visual data or combinations of sounds in auditory data. For example, higher layers can learn to represent complex shapes in visual data or words in auditory data. Even higher layers can learn to recognize common visual objects or spoken phrases.

[0062] Deep learning architectures perform particularly well when applied to problems with a natural hierarchical structure. For example, the classification of motorized vehicles can benefit from first learning to identify features such as wheels, windshields, and others. These features can then be combined in different ways at higher levels to identify cars, trucks, and airplanes.

[0063] Neural networks can be designed to have multiple connectivity patterns. In feedforward networks, information is passed from lower layers to higher layers, where each neuron in a given layer communicates with neurons in higher layers. As described above, hierarchical representations can be built in successive layers of a feedforward network. Neural networks can also have recurrent or feedback (also known as top-down) connections. In recurrent connections, the output from a neuron in a given layer can be passed to another neuron in the same layer. Recurrent architectures can help identify patterns across more than one block of input data that is sequentially delivered to the neural network. Connections from neurons in a given layer to neurons in lower layers are called feedback (or top-down) connections. Networks with many feedback connections can be helpful when the recognition of higher-level concepts can aid in discerning specific lower-level features of the input.

[0064] Figure 7 Two sets of images 700 are provided, illustrating the forward diffusion process (which is fixed) and the reverse diffusion process (which is learned) of the diffusion model. Figure 7 As shown in the forward diffusion process, noise 703 is gradually added to the first set of images 702 at different time steps in a total of T time steps (e.g., forming a Markov chain), thereby generating a series of noisy samples X1 to XT.

[0065] From a training perspective, the diffusion model acquires an image and slowly adds noise to it to destroy the information within the image. In some respects, the noise 703 is Gaussian noise. Each time step can correspond to... Figure 7 Each consecutive image in the first set of images 702 shown. Figure 7 The initial image X0 is an image of a cat. Adding noise 703 to each image (corresponding to noisy samples X1 to XT) causes the pixels in each image to gradually diffuse until the final image (corresponding to sample XT) substantially matches the noise distribution. For example, by adding noise, as the time step increases, each data sample X1 to XT gradually loses its distinguishable features, eventually causing the final sample XT to be equivalent to the target noise distribution, such as a Gaussian with unit variance of zero. .

[0066] The second set of images, 704, illustrates the reverse diffusion process, where XT is the starting point of a noisy image (e.g., an image with Gaussian noise). The diffusion model can be trained to reverse the diffusion process (e.g., by training the model pθ(xt-1 |xt)) to generate new data. In some aspects, the diffusion model can be trained by finding the inverse Markov transformation that maximizes the likelihood of the training data. By traversing backwards along the time-step chain, the diffusion model can generate new data. For example, as... Figure 7 As shown, the reverse diffusion process continues to generate an image of X0 as the vase. In other cases, the input and output data may vary based on the task for which the diffusion model was trained.

[0067] As noted above, the diffusion model is trained to denoise or restore the original image X0 in a progressive process, as shown in the second set of images 704. In some respects, the neural network of the diffusion model can be trained to recover Xt given Xt-1, as provided in the following example equation: .

[0068] The diffusion nucleus can be defined as follows: definition

[0069] Sampling can be defined as follows:

[0070] In some cases, Value scheduling (also known as noise scheduling) is designed to make and .

[0071] The diffusion model runs iteratively to progressively generate the input image X0. In one example, the model may have twenty steps. However, in other examples, the number of steps can vary.

[0072] Figure 8 This is an illustration of how a diffusion model can be used to distribute diffusion data from initial data to noise in the forward diffusion direction, based on some aspects. Note that the initial data q(X0) is detailed in the initial stage of the diffusion process. An illustrative example of data q(X0) is... Figure 7 The initial image of the vase is shown. As the diffusion model iterates and sampling noise is added to the data iteratively from t=0 to t=T, as... Figure 8 As shown, the data becomes noisier and may eventually result in pure noise (e.g., in q(X)). T ) place). Figure 8 The example illustrates the progress of the data and how the data spreads along with noise during the forward diffusion process.

[0073] In some respects, the distribution of diffuse data (e.g., such as...) Figure 8 (As shown) can be as follows: .

[0074] In the above equation, Indicates the distribution of diffusion data. Indicates the joint distribution. This represents the distribution of the input data, and It is a diffusion kernel. In this respect, the model can be improved by first sampling... And then sample To sample (This can be called ancestor sampling). The diffusion kernel takes input and returns a vector or other data structure as output.

[0075] The following is an overview of the training and sampling algorithms for the diffusion model. The training algorithm may include the following steps: repeat Perform gradient descent steps on the following expression Until convergence The sampling algorithm may include the following steps: for Finish return

[0076] Figure 9 This is a diagram 900 illustrating a U-Net architecture for a diffusion model based on some aspects. An initial image 902 (e.g., of a cat) is provided to the U-Net architecture 900, which includes a series of Residual Network (ResNet) blocks and self-attention layers to represent the network. (xt, t). The U-Net architecture 900 also includes a fully connected layer 908. In some cases, the time representation 910 can be a sinusoidal localization embedding or a random Fourier feature. The noise output 906 from the forward diffusion process is also shown.

[0077] U-Net architecture 900 includes, for example, Figure 9 The diagram shows a contraction path 904 and an expansion path 905, which gives it a U-shaped architecture. The contraction path 904 can be a convolutional network comprising repeated convolutional layers (which apply convolutional operations), each followed by a rectified linear unit (ReLU) and max-pooling operation. When processing an image (e.g., image 902) during the contraction path 904, the spatial information of image 902 is reduced as features are generated. The expansion path 905 combines features and spatial information through a series of up-convolutions and cascading with high-resolution features from the contraction path 904. Some layers can be self-attention layers, which explicitly model complete contextual information by leveraging global interactions between semantic features at the encoder ends.

[0078] Figure 10 This is an exemplary example of a neural network 1000 (e.g., a deep learning neural network) that can be used to implement machine learning-based image generation, feature segmentation, implicit neural representation generation, rendering, classification, object detection, image recognition (e.g., face recognition, object recognition, scene recognition, etc.), feature extraction, authentication, gaze detection, gaze prediction, and / or automation.

[0079] Input layer 1002 includes input data. Neural network 1000 includes multiple hidden layers 1006a, 1006b through 1006n. Hidden layers 1006a, 1006b through 1006n comprise "n" hidden layers, where "n" is an integer greater than or equal to one. Multiple hidden layers can be made to include as many layers as needed for a given application. Neural network 1000 also includes an output layer 1004, which provides the output produced by the processing performed by hidden layers 1006a, 1006b through 1006n.

[0080] The neural network 1000 may be or may include a multi-layer neural network with interconnected nodes. Each node may represent a piece of information. The information associated with these nodes is shared between different layers, and each layer retains the information while processing it. In some cases, the neural network 1000 may include a feedforward network, in which case there are no feedback connections in which the network's output is fed back into itself. In some cases, the neural network 1000 may include a recurrent neural network, which may have loops that allow information to be carried across nodes when reading from the input.

[0081] Information can be exchanged between nodes through node-to-node interconnects between layers. Nodes in input layer 1002 can activate the node set in the first hidden layer 1006a. For example, as shown, each input node in input layer 1002 is connected to each node in the first hidden layer 1006a. Nodes in the first hidden layer 1006a can transform the information of each input node by applying an activation function to the input node information. The information derived from this transformation can then be passed to nodes in the next hidden layer 1006b, activating those nodes, which can then perform their own specified functions. Example functions include convolution, upsampling, data transformation, and / or any other suitable functions. The output of hidden layer 1006b can then activate nodes in the next hidden layer, and so on. The output of the last hidden layer 1006n can activate one or more nodes in output layer 1004, at which the output is provided. In some cases, although a node in neural network 1000 (e.g., node 1008) is shown as having multiple output lines, the node has a single output and is shown as all lines output from the node representing the same output value.

[0082] In some cases, each node or the interconnection between nodes may have weights, which are a set of parameters derived from the training of the neural network 1000. Once the neural network 1000 is trained, it can be called a trained neural network, which can be used to perform one or more operations. For example, the interconnection between nodes may represent a piece of information about what the interconnected nodes have learned. This interconnection may have tunable numerical weights that can be tuned (e.g., based on the training dataset), allowing the neural network 1000 to adapt to the input and learn as more and more data is processed.

[0083] The neural network 1000 can be pre-trained to process features from the data in the input layer 1002 using different hidden layers 1006a, 1006b to 1006n, so as to provide an output through the output layer 1004. In an example where the neural network 1000 is used to identify features in an image, the neural network 1000 can be trained using training data that includes both images and labels, as described above. For example, training images can be input into the network, where each training image has a label indicating features in the image (for feature segmentation machine learning systems) or a label indicating the category of activity in each image. In an example where object classification is used for illustrative purposes, the training images may include images of the number 2, in which case the label of the image may be [0 0 1 0 0 0 0 0 0 0].

[0084] In some cases, the Neural Network 1000 can use a training process called backpropagation to adjust the weights of its nodes. As noted above, the backpropagation process can include forward pass, loss function, back pass, and weight update. For each training iteration, forward pass, loss function, back pass, and parameter update are performed. For each set of training images, this process can be repeated up to a certain number of iterations until the Neural Network 1000 is trained well enough to accurately tune the weights of each layer.

[0085] For an example of identifying objects in an image, the forward pass may include passing a training image through a neural network 1000. The weights are initially randomized before training the neural network 1000. As an illustrative example, the image may include a numerical array representing the pixels of the image. Each number in the array may include a value from 0 to 255 describing the intensity of the pixel at that location in the array. In one example, the array may include a 28×28×3 numerical array with 28 rows and 28 columns of pixels and 3 color components (such as red, green, and blue, or lightness and two chroma components, etc.).

[0086] As noted above, for the first training iteration of a neural network 1000, the output may include values ​​due to the weights being randomly selected during initialization without prioritizing any particular class. For example, if the output is a vector with probabilities that an object includes different classes, the probability values ​​for each class may be equal or at least very similar (e.g., for ten possible classes, each class may have a probability value of 0.1). With the initial weights, the neural network 1000 cannot determine low-level features and therefore cannot make an accurate determination of what the object's classification might be. A loss function can be used to analyze the error in the output. Any suitable loss function can be defined, such as cross-entropy loss. Another example of a loss function includes mean squared error (MSE), which is defined as... The loss can be set to equal E. 总计 The value of .

[0087] For the first training image, the loss (or error) will be high because the actual value will be significantly different from the predicted output. The goal of training is to minimize the loss so that the predicted output matches the training labels. The Neural Network 1000 performs backpropagation by determining which inputs (weights) contribute most to the network's loss and can adjust the weights to reduce and eventually minimize the loss. The derivative of the loss with respect to the weights (denoted as dL / dW, where W is the weight at a specific layer) can be calculated to determine the weights that contribute most to the network's loss. After calculating the derivative, a weight update can be performed by updating all the weights of the filter. For example, the weights can be updated so that they change in the opposite direction of the gradient. A weight update can be represented as... Where w represents the weight, w i Let represent the initial weights, and η represent the learning rate. The learning rate can be set to any suitable value, where a high learning rate includes larger weight updates, while a lower value indicates smaller weight updates.

[0088] Neural Network 1000 can include any suitable deep network. An example includes a Convolutional Neural Network (CNN), which includes an input layer and an output layer, with multiple hidden layers between them. The hidden layers of a CNN include a series of convolutional layers, non-linear layers, pooling layers (for downsampling), and fully connected layers. Neural Network 1000 can include any other deep network besides CNNs, such as autoencoders, deep belief networks (DBNs), recurrent neural networks (RNNs), etc.

[0089] Figure 11 This is an exemplary example of a Convolutional Neural Network (CNN) 1100. The input layer 1102 of the CNN 1100 includes data representing an image or frame. For example, the data may include a numerical array representing pixels of an image, where each number in the array includes a value from 0 to 255 describing the pixel intensity at that location in the array. Using the previous example above, the array may include a 28×28×3 numerical array with 28 rows and 28 columns of pixels and 3 color components (e.g., red, green, and blue, or lightness and two chroma components, etc.). The image may be passed through a convolutional hidden layer 1104, an optional non-linear activation layer, a pooling hidden layer 1106, and a fully connected layer 1108 (which may be hidden) to obtain an output at the output layer 1110. Although... Figure 11 Only one hidden layer from each hidden layer is shown in the diagram, but those skilled in the art will understand that multiple convolutional hidden layers, non-linear layers, pooling hidden layers, and / or fully connected layers may be included in the CNN 1100. As previously described, the output may indicate a single category of an object, or may include probabilities that best describe the category of an object in an image.

[0090] The first layer of CNN 1100 can be a convolutional hidden layer 1104. The convolutional hidden layer 1104 analyzes the image data from the input layer 1102. Each node in the convolutional hidden layer 1104 is connected to a region of the input image called a receptive field (pixel). The convolutional hidden layer 1104 can be thought of as one or more filters (each filter corresponding to a different activation or feature map), where each convolutional iteration of the filter is a node or neuron in the convolutional hidden layer 1104. For example, the region of the input image covered by the filter at each convolutional iteration will be the receptive field of the filter. In an exemplary example, if the input image comprises a 28×28 array and each filter (and its corresponding receptive field) is a 5×5 array, then there will be 24×24 nodes in the convolutional hidden layer 1104. Each connection between a node and its receptive field learns weights and, in some cases, learns an overall bias, such that each node learns to analyze its specific local receptive field in the input image. Each node in the convolutional hidden layer 1104 will have the same weights and biases (called shared weights and shared biases). For example, the filter has a weight (digital) array and the same depth as the input. For the image frame example, the filter would have a depth of 3 (based on the three color components of the input image). An exemplary example of the filter array size is 5×5×3, corresponding to the size of the receptive field of a node.

[0091] The convolutional property of the convolutional hidden layer 1104 is due to the fact that each node of the convolutional layer is applied to its corresponding receptive field. For example, the filter of the convolutional hidden layer 1104 may begin at the top left corner of the input image array and may convolve around the input image. As noted above, each convolutional iteration of the filter can be considered as a node or neuron of the convolutional hidden layer 1104. In each convolutional iteration, the value of the filter is multiplied by the corresponding number of original pixel values ​​of the image (e.g., a 5×5 filter array is multiplied by a 5×5 array of input pixel values ​​at the top left corner of the input image array). The multiplications from each convolutional iteration can be summed to obtain the sum of that iteration or node. Next, the process continues at the next position in the input image based on the receptive field of the next node in the convolutional hidden layer 1104. For example, the filter may move a step size (called stride) to the next receptive field. The stride may be set to 1 or any other suitable amount. For example, if the stride is set to 1, the filter will move 1 pixel to the right in each convolutional iteration. Processing the filter at each unique location in the input volume produces a number representing the filter result at that location, thereby determining a sum value for each node of the convolutional hidden layer 1104.

[0092] The mapping from the input layer to the convolutional hidden layer 1104 is called an activation map (or feature map). An activation map includes node-specific values ​​representing the filter results at each location of the input volume. Activation maps may include arrays containing various sums of values ​​produced by the filter for each iteration of the input volume. For example, if a 5×5 filter is applied to each pixel of a 28×28 input image (with a stride of 1), the activation map would consist of a 24×24 array. The convolutional hidden layer 1104 may include several activation maps to identify multiple features in the image. Figure 11 The example shown includes three activation maps. Using the three activation maps, the convolutional hidden layer 1104 can detect three different types of features, each of which is detectable across the entire image.

[0093] In some examples, nonlinear hidden layers can be applied after convolutional hidden layer 1104. Nonlinear layers can be used to introduce nonlinearity into a system that has already computed linear operations. An exemplary example of a nonlinear layer is the Corrected Linear Unit (ReLU) layer. A ReLU layer applies the function f(x) = max(0, x) to all values ​​in the input volume, which changes all negative activations to 0. Therefore, ReLU can add nonlinearity to CNN 1100 without affecting the receptive field of convolutional hidden layer 1104.

[0094] A pooling hidden layer 1106 can be applied after the convolutional hidden layer 1104 (and, in use, after the non-linear hidden layer). The pooling hidden layer 1106 is used to simplify the information in the output of the convolutional hidden layer 1104. For example, the pooling hidden layer 1106 can take each activation map output from the convolutional hidden layer 1104 and use a pooling function to generate a condensed activation map (or feature map). Max pooling is an example of a function performed by the pooling hidden layer. The pooling hidden layer 1106 uses other forms of pooling functions, such as average pooling, L2 norm pooling, or other suitable pooling functions. Pooling functions (e.g., max pooling filters, L2 norm filters, or other suitable pooling filters) are applied to each activation map included in the convolutional hidden layer 1104. Figure 11 In the example shown, three pooling filters are used to convolve the three activation maps in the hidden layer 1104.

[0095] In some examples, max pooling can be used by applying a max pooling filter (e.g., of size 2×2) with a stride (e.g., equal to the dimension of the filter, such as stride 2) to the activation map output from convolutional hidden layer 1104. The output from the max pooling filter includes the maximum number in each sub-region of the filter convolution. Using a 2×2 filter as an example, each unit in the pooling layer summarizes a region of 2×2 nodes from the previous layer (each node is a value in the activation map). For example, four values ​​(nodes) in the activation map will be analyzed by the 2×2 max pooling filter at each iteration of the filter, with the maximum of the four values ​​being output as the "maximum" value. If such a max pooling filter is applied to an activation filter of 24×24 nodes from convolutional hidden layer 1104, the output from pooling hidden layer 1106 will be an array of 12×12 nodes.

[0096] In some examples, L2 norm pooling filters may also be used. L2 norm pooling filters involve calculating the square root of the sum of squares of the values ​​in a 2×2 region (or other suitable region) of the activation map (instead of calculating the maximum value as done in max pooling), and using the calculated value as the output.

[0097] Pooling functions (e.g., max pooling, L2 norm pooling, or other pooling functions) determine whether a given feature is found anywhere in a region of the image and discard the exact location information. This can be done without affecting the results of feature detection because once a feature has been found, its exact location is less important than its approximate location relative to other features. Max pooling (and other pooling methods) offers the benefit of having far fewer pooling features, thus reducing the number of parameters required in subsequent layers of the CNN 1100.

[0098] The final connection in the network is a fully connected layer that connects each node from the pooling hidden layer 1106 to each output node in the output layer 1110. Using the example above, the input layer comprises 28×28 nodes encoding the pixel intensity of the input image, the convolutional hidden layer 1104 comprises 3×24×24 hidden feature nodes based on applying a 5×5 local receptive field (for filtering) to three activation maps, and the pooling hidden layer 1106 comprises a layer of 3×12×12 hidden feature nodes based on applying a max-pooling filter to a 2×2 region across each of the three feature maps. Extending this example, the output layer 1110 may comprise ten output nodes. In this example, each node of the 3×12×12 pooling hidden layer 1106 is connected to each node of the output layer 1110.

[0099] The fully connected layer 1108 takes the output of the previous pooling hidden layer 1106 (which should represent an activation map of high-level features) and determines the features most relevant to a particular class. For example, the fully connected layer 1108 can determine the high-level features most relevant to a particular class and may include weights (nodes) for the high-level features. The product between the weights of the fully connected layer 1108 and the pooling hidden layer 1106 can be computed to obtain the probabilities for different classes. For example, if the CNN 1100 is used to predict that the object in the image is a person, there will be high values ​​in the activation map representing the high-level features of a person (e.g., two legs, a face at the top of the object, two eyes at the top left and top right of the face, a nose in the middle of the face, a mouth at the bottom of the face, and / or other features common to people).

[0100] In some examples, the output from output layer 1110 may include an M-dimensional vector (M=10 in the previous example). M indicates the number of classes from which CNN 1100 must choose when classifying objects in an image. Other example outputs may also be provided. Each number in the M-dimensional vector represents the probability that an object belongs to a certain class. In an exemplary example, if the 10-dimensional output vector representing objects of ten different classes is [0 0 0.05 0.8 0 0.15 0 0 0 0], then the vector indicates a 5% probability that the image is an object of the third class (e.g., a dog), an 80% probability that the image is an object of the fourth class (e.g., a person), and a 15% probability that the image is an object of the sixth class (e.g., a kangaroo). The probability of a class can be considered as the confidence level that an object is part of that class.

[0101] Figure 12 An example computing device architecture 1200 is illustrated, illustrating example computing devices that can implement the various technologies described herein. In some examples, the computing device may include a mobile device, a wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a personal computer, a laptop computer, a video server, a vehicle (or a computing device within a vehicle), or other devices. Components of the computing device architecture 1200 are shown to communicate electrically with each other using a connection 1205, such as a bus. The example computing device architecture 1200 includes a processing unit (CPU or processor) 1210 and a computing device connection 1205 that couples various computing device components, including computing device memories 1215 (such as read-only memory (ROM) 1220 and random access memory (RAM) 1225), to the processor 1210.

[0102] The computing device architecture 1200 may include a cache of high-speed memory that is directly connected to, very close to, or integrated as part of the processor 1210. The computing device architecture 1200 may copy data from memory 1215 and / or storage device 1230 to cache 1212 for fast access by the processor 1210. In this way, the cache can provide performance improvements by avoiding latency for the processor 1210 while waiting for data. These and other modules may control or be configured to control the processor 1210 to perform various actions. Other computing device memory 1215 may also be used. Memory 1215 may include various different types of memory with different performance characteristics. The processor 1210 may include any general-purpose processor and hardware or software services configured to control the processor 1210 (such as services 11232, 1234, and 1236 stored in storage device 1230), as well as dedicated processors in which software instructions are incorporated into the processor design. The processor 1210 may be a self-contained system containing multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors can be symmetric or asymmetric.

[0103] To enable user interaction with the computing device architecture 1200, the input device 1245 can represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, voice input, etc. The output device 1235 can also be one or more of a variety of output mechanisms known to those skilled in the art, such as a display, projector, television, speaker equipment, etc. In some instances, a multi-mode computing device allows the user to provide multiple types of input to communicate with the computing device architecture 1200. The communication interface 1240 typically controls and manages user input and computing device output. There are no limitations on operation on any particular hardware arrangement, and therefore the underlying features here can be easily replaced to obtain improved hardware or firmware arrangements as they are developed.

[0104] Storage device 1230 is a non-volatile memory and may be a hard disk or other type of computer-readable medium capable of storing computer-accessible data, such as a magnetic tape cassette, flash memory card, solid-state memory device, digital multifunction disk, magnetic tape cartridge, random access memory (RAM) 1225, read-only memory (ROM) 1220, and hybrid forms thereof. Storage device 1230 may include services 1232, 1234, 1236 for controlling processor 1210. Other hardware or software modules are envisioned. Storage device 1230 may be connected to computing device connection 1205. In one aspect, a hardware module performing a specific function may include software components stored in a computer-readable medium connected to necessary hardware components, such as processor 1210, connection 1205, output device 1235, etc., to perform that function.

[0105] Various aspects of this disclosure are applicable to any suitable electronic device (such as a security system, smartphone, tablet, laptop, vehicle, drone, or other device) that includes or is coupled to one or more active depth sensing systems. Although the following description relates to devices having or coupled to a light projector, various aspects of this disclosure are applicable to devices having any number of light projectors and are therefore not limited to any particular device.

[0106] The term "device" is not limited to one or a specific number of physical objects (such as a smartphone, a controller, a processing system, etc.). As used herein, a device can be any electronic device having one or more parts that implement at least some parts of this disclosure. Although the following description and examples use the term "device" to describe various aspects of this disclosure, the term "device" is not limited to a specific configuration, type, or number of objects. Additionally, the term "system" is not limited to multiple components or a particular implementation. For example, a system may be implemented on one or more printed circuit boards or other substrates and may have movable or static components. Although the following description and examples use the term "system" to describe various aspects of this disclosure, the term "system" is not limited to a specific configuration, type, or number of objects.

[0107] Specific details have been provided in the foregoing description to offer a thorough understanding of the embodiments and examples presented herein. However, those skilled in the art will understand that embodiments can be practiced without these specific details. For clarity, in some instances, the technology may be presented as comprising individual functional blocks, including functional blocks containing devices, device components, steps or routines in methods embodied in software or a combination of hardware and software. Additional components may be used in addition to those shown in the figures and / or described herein. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form to avoid obscuring these embodiments with unnecessary detail. In other cases, well-known circuits, processes, algorithms, structures, and techniques may be shown without necessary detail to avoid obscuring the embodiments.

[0108] Individual implementations may be described above as processes or methods depicted as flowcharts, flow diagrams, data flow diagrams, structure diagrams, or block diagrams. Although flowcharts may describe operations as sequential processes, many operations within an operation may be executed in parallel or concurrently. Furthermore, the order of operations may be rearranged. A process terminates when its operations are completed, but a process may have additional steps not included in the accompanying drawings. A process may correspond to a method, function, procedure, subroutine, subroutine, etc. When a process corresponds to a function, the termination of the process may correspond to the function returning to the calling function or the main function.

[0109] The processes and methods described in the examples above can be implemented using stored computer-executable instructions or computer-executable instructions otherwise obtainable from a computer-readable medium. Such instructions may include, for example, instructions and data that configure, cause or otherwise configure, a general-purpose computer, special-purpose computer, or processing device to perform a function or group of functions. The portion of the computer resources used may be accessible via a network. Computer-executable instructions may be, for example, binary files, intermediate format instructions (such as assembly language), firmware, source code, etc.

[0110] The term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing, or carrying instructions and / or data. Computer-readable media may include non-transitory media in which data can be stored and which do not include carrier waves and / or transient electronic signals propagating wirelessly or over a wired connection. Examples of non-transitory media may include, but are not limited to, magnetic disks or magnetic tapes, optical storage media (such as flash memory), memory or memory devices, magnetic disks or optical discs, flash memory, USB devices provided with non-volatile memory, network storage devices, compressed optical discs (CDs) or digital versatile optical discs (DVDs), any suitable combinations thereof, etc. Computer-readable media may store code and / or machine-executable instructions thereon, which may represent procedures, functions, subroutines, programs, routines, subroutines, modules, software packages, classes, or any combination of instructions, data structures, or program statements. Code segments may be coupled to other code segments or hardware circuitry by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, independent variables, parameters, data, etc., can be transmitted, forwarded, or sent through any suitable means, including memory sharing, message passing, token passing, network sending, etc.

[0111] In some implementations, computer-readable storage devices, media, and memories may include wired or wireless signals containing bit streams, etc. However, when referred to, non-transitory computer-readable storage media explicitly exclude media such as energy, carrier signals, electromagnetic waves, and the signals themselves.

[0112] Devices implementing the processes and methods according to these disclosures may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented as software, firmware, middleware, or microcode, program code or code segments (e.g., computer program products) for performing necessary tasks may be stored in a computer-readable or machine-readable medium. A processor may perform the necessary tasks. Typical examples of form factors include laptop computers, smartphones, mobile phones, tablet devices, or other small form factor personal computers, personal digital assistants, rack-mounted devices, standalone devices, etc. The functionality described herein may also be embodied in peripheral devices or interlocking cards. By further example, such functionality may also be implemented on circuit boards of different chips or different processes executed on a single device.

[0113] Instructions, media for delivering such instructions, computing resources for executing them, and other structures for supporting such computing resources are example components for providing the functionality described in this disclosure.

[0114] In the foregoing description, various aspects of this application have been described with reference to specific embodiments thereof; however, those skilled in the art will recognize that this application is not limited thereto. Therefore, although exemplary embodiments of this application have been described in detail herein, it is to be understood that the inventive concept may be embodied and adopted in various other ways, and the appended claims are intended to be construed as including such variations, unless limited by prior art. The various features and aspects of the applications described above may be used individually or in combination. Furthermore, without departing from the broader spirit and scope of this specification, the embodiments may be used in any number of environments and applications beyond those described herein. Therefore, the specification and drawings should be considered illustrative rather than restrictive. For illustrative purposes, the methods are described in a particular order. It should be understood that in alternative embodiments, the methods may be performed in a different order than described.

[0115] Those skilled in the art will understand that, without departing from the scope of this description, the symbols or terms less than ("<") and greater than (">") used herein can be represented by less than or equal to ("<"), respectively. ") and greater than or equal to (" The symbol ) is used instead.

[0116] When a component is described as being "configured" to perform certain operations, such a configuration can be achieved, for example, by designing electronic circuits or other hardware to perform the operations, by programming programmable electronic circuits (e.g., microprocessors or other suitable electronic circuits) to perform the operations, or any combination thereof.

[0117] The phrase "coupled to" means any component is physically connected directly or indirectly to another component, and / or any component communicates directly or indirectly with another component (e.g., connected to another component via a wired or wireless connection and / or other suitable communication interface).

[0118] The claim language or other language that states "at least one of" and / or "one or more of" in a set indicates that one member of the set or multiple members of the set (in any combination) satisfies the claim. For example, the claim language that states "at least one of A and B" or "at least one of A or B" means A, B, or A and B. In another example, the claim language that states "at least one of A, B, and C" or "at least one of A, B, or C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language "at least one of" and / or "one or more of" in a set does not limit the set to the items listed in the set. For example, the claim language that states "at least one of A and B" or "at least one of A or B" may mean A, B, or A and B, and may additionally include items not listed in the set of A and B.

[0119] Claim language or other languages ​​that state "at least one processor, at least one processor is configured to," "at least one processor is configured to," "one or more processors, one or more processors are configured to," "one or more processors are configured to," etc., indicate that one or more processors (in any combination) can perform the associated operations. For example, claim language that states "at least one processor, at least one processor is configured to: X, Y, and Z" means that a single processor can be used to perform operations X, Y, and Z; or that multiple processors are each assigned a specific subset of the task of operations X, Y, and Z, such that the multiple processors together perform X, Y, and Z; or that a group of multiple processors work together to perform operations X, Y, and Z. In another example, claim language that states "at least one processor, at least one processor is configured to: X, Y, and Z" could mean that any single processor can perform only at least one subset of operations X, Y, and Z.

[0120] When referring to one or more elements that perform functions (e.g., steps of a method), one element may perform all functions, or more than one element may jointly perform these functions. When more than one element jointly performs these functions, each function does not need to be performed by every single element (e.g., different functions may be performed by different elements), and / or each function does not need to be performed by only one element as a whole (e.g., different elements may perform different sub-functions of a function). Similarly, when referring to one or more elements configured to cause another element (e.g., a device) to perform functions, one element may be configured to cause another element to perform all functions, or more than one element may be jointly configured to cause another element to perform these functions.

[0121] When referring to an entity that performs or is configured to perform functions (e.g., steps of a method) (e.g., any entity or device described herein), the entity may be configured to cause one or more elements (individually or collectively) to perform those functions. One or more components of the entity may include at least one memory, at least one processor, at least one communication interface, another component configured to perform one or more of those functions, and / or any combination thereof. When referring to an entity that performs functions, the entity may be configured to cause one component to perform all functions, or to cause more than one component to perform those functions collectively. When the entity is configured to cause more than one component to perform those functions collectively, each function does not need to be performed by every single component (e.g., different functions may be performed by different components), and / or each function does not need to be performed by only one component as a whole (e.g., different components may perform different sub-functions of a function).

[0122] The various exemplary logic blocks, modules, circuits, and algorithm steps described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware, computer software, firmware, or combinations thereof. To clearly illustrate this interchangeability between hardware and software, various exemplary components, blocks, modules, circuits, and steps have been broadly described above in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in different ways for each specific application, but such implementation decisions should not be construed as departing from the scope of this application.

[0123] The techniques described herein can also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques can be implemented in any of a variety of devices, such as general-purpose computers, wireless communication devices (mobile phones), or integrated circuit devices with multiple uses, including applications in wireless communication devices (mobile phones) and other devices. Any feature described as a module or component can be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, these techniques can be implemented at least in part by a computer-readable data storage medium comprising program code including instructions that, when executed, perform one or more of the methods described above. The computer-readable data storage medium can form part of a computer program product, which may include packaging materials. The computer-readable medium may include memory or data storage media, such as random access memory (RAM) (such as synchronous dynamic random access memory (SDRAM)), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic or optical data storage media, etc. Additionally or alternatively, the technology may be implemented at least in part by a computer-readable communication medium that carries or conveys program code in the form of instructions or data structures that can be accessed, read and / or executed by a computer, such as propagated signals or waves.

[0124] The program code can be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Such processors can be configured to perform any of the techniques described in this disclosure. A general-purpose processor may be a microprocessor; however, in alternatives, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. Therefore, as used herein, the term "processor" may refer to any of the foregoing structures, any combination of the foregoing structures, or any other structure or means suitable for implementing the techniques described herein.

[0125] The exemplary aspects of this disclosure include: Aspect 1. An apparatus for adapting a pre-trained machine learning model, the apparatus comprising: one or more memories configured to store training images; and one or more processors coupled to the one or more memories and configured to: encode the training images into a first feature vector, the training images including a first object located at a first position; generate a second feature vector using a set of weights based on a set of sinusoidal functions; combine the first feature vector and the second feature vector to generate a combined feature vector; process the combined feature vector using a visual language model to obtain a second position of the first object; and adjust the set of weights based on a comparison between the first position and the second position.

[0126] Aspect 2. The apparatus according to aspect 1, wherein the set of sinusoidal functions includes cosine functions and sine functions.

[0127] Aspect 3. The apparatus according to any one of Aspects 1 to 2, wherein the second position includes the text coordinates of the bounding box.

[0128] Aspect 4. The apparatus according to any one of aspects 1 to 3, wherein the combined feature vector and textual cues are processed using the visual language model to obtain the second position of the first object.

[0129] Aspect 5. The apparatus according to aspect 4, wherein the text prompt includes a prompt regarding the location of the first object.

[0130] Aspect 6. The apparatus according to any one of Aspects 1 to 5, wherein the one or more processors are further configured to generate the training images.

[0131] Aspect 7. The apparatus according to aspect 6, wherein, in order to generate the training image, the one or more processors are configured to insert the first object into a background image.

[0132] Aspect 8. The apparatus according to aspect 7, wherein, in order to generate the training image, the one or more processors are configured to generate an image of the first object.

[0133] Aspect 9. The apparatus according to aspect 8, wherein, in order to generate the image of the first object, the one or more processors are configured to submit an object prompt to a text-to-image generation model.

[0134] Aspect 10. The apparatus according to any one of Aspects 6 to 8, wherein the weights of the visual language model are frozen.

[0135] Aspect 11. The apparatus according to any one of Aspects 1 to 10, wherein the at least one processor is configured to adjust the weight set by on-device training based on a machine learning system.

[0136] Aspect 12. An apparatus for image processing, the apparatus comprising: one or more memories configured to store an image; and one or more processors coupled to the one or more memories and configured to: process the image and a first text cue using a visual language model to obtain a list of objects in the image; process the image and a second text cue using an adapted visual language model, the second text cue including a first object from the list of objects; and receive a set of coordinates of the first object in the image.

[0137] Aspect 13. The apparatus according to aspect 12, wherein the first text prompt includes a prompt for a list of objects in the image.

[0138] Aspect 14. The apparatus according to any one of Aspects 12 to 13, wherein the second text prompt includes a prompt regarding the location of the first object.

[0139] Aspect 15. The apparatus according to any one of aspects 11 to 14, wherein the visual language model and the adapted visual language model are the same machine language model.

[0140] Aspect 16. The apparatus according to any one of aspects 11 to 15, wherein the weights of the visual language model are frozen.

[0141] Aspect 17. The apparatus according to any one of aspects 11 to 16, the apparatus further comprising a camera configured to capture the image.

[0142] Aspect 18. A method for adapting a pre-trained machine learning model, the method comprising: encoding a training image into a first feature vector, the training image including a first object located at a first position; generating a second feature vector using a set of weights based on a set of sinusoidal functions; combining the first feature vector and the second feature vector to generate a combined feature vector; processing the combined feature vector using a visual language model to obtain a second position of the first object; and adjusting the set of weights based on a comparison between the first position and the second position.

[0143] Aspect 19. The method according to aspect 18, wherein the set of sinusoidal functions includes cosine functions and sine functions.

[0144] Aspect 20. The method according to any one of Aspects 18 to 19, wherein the second position includes the text coordinates of the bounding box.

[0145] Aspect 21. The method according to any one of aspects 18 to 20, wherein the combined feature vector and textual cues are processed using the visual language model to obtain the second position of the first object.

[0146] Aspect 22. The method according to aspect 21, wherein the text prompt includes a prompt regarding the location of the first object.

[0147] Aspect 23. The method according to any one of aspects 18 to 22, the method further comprising generating the training image.

[0148] Aspect 24. The method according to aspect 23, wherein generating the training image includes inserting the first object into a background image.

[0149] Aspect 25. The method according to aspect 24, wherein generating the training image further includes generating an image of the first object.

[0150] Aspect 26. The method according to aspect 25, wherein the image of the first object is generated by submitting object prompts to a text-to-image generation model.

[0151] Aspect 27. The method according to any one of Aspects 23 to 26, wherein the weights of the visual language model are frozen.

[0152] Aspect 28. A method for image processing, the method comprising: processing an image and a first text prompt using a visual language model to obtain a list of objects in the image; processing the image and a second text prompt using an adapted visual language model, the second text prompt including a first object from the list of objects; and receiving a set of coordinates of the first object in the image.

[0153] Aspect 29. The method according to aspect 28, wherein the first text prompt includes a prompt for a list of objects in the image.

[0154] Aspect 30. The method according to any one of Aspects 28 to 29, wherein the second text prompt includes a prompt regarding the location of the first object.

[0155] Aspect 31. The method according to any one of Aspects 28 to 30, wherein the visual language model and the adapted visual language model are the same machine language model.

[0156] Aspect 32. The method according to any one of Aspects 28 to 31, wherein the weights of the visual language model are frozen.

[0157] Aspect 33. A non-transitory computer-readable medium having instructions stored thereon, the instructions causing the one or more processors, when executed, to: encode a training image into a first feature vector, the training image including a first object located at a first position; generate a second feature vector using a set of weights based on a set of sinusoidal functions; combine the first feature vector with the second feature vector to generate a combined feature vector; process the combined feature vector using a visual language model to obtain a second position of the first object; and adjust the set of weights based on a comparison between the first position and the second position.

[0158] Aspect 34. The non-transitory computer-readable medium according to aspect 33, wherein the set of sinusoidal functions includes cosine functions and sine functions.

[0159] Aspect 35. The non-transitory computer-readable medium according to any one of Aspects 33 to 34, wherein the second location includes the text coordinates of a bounding box.

[0160] Aspect 36. A non-transitory computer-readable medium according to any one of aspects 33 to 35, wherein the combined feature vector and textual cues are processed using the visual language model to obtain the second position of the first object.

[0161] Aspect 37. The non-transitory computer-readable medium according to aspect 35, wherein the text prompt includes a prompt regarding the location of the first object.

[0162] Aspect 38. A non-transitory computer-readable medium according to any one of aspects 33 to 37, wherein the instructions cause the one or more processors to generate the training image.

[0163] Aspect 39. The non-transitory computer-readable medium according to aspect 38, wherein, in order to generate the training image, the instructions cause the one or more processors to insert the first object into a background image.

[0164] Aspect 40. The non-transitory computer-readable medium according to aspect 39, wherein, in order to generate the training image, the instructions cause the one or more processors to generate an image of the first object.

[0165] Aspect 41. The non-transitory computer-readable medium according to aspect 40, wherein, in order to generate the image of the first object, the instructions cause the one or more processors to submit an object prompt to a text-to-image generation model.

[0166] Aspect 42. The non-transitory computer-readable medium according to any one of aspects 38 to 41, wherein the weights of the visual language model are frozen.

[0167] Aspect 43. A non-transitory computer-readable medium having instructions stored thereon, the instructions, when executed by one or more processors, causing the one or more processors to: process an image and a first text prompt using a visual language model to obtain a list of objects in the image; process the image and a second text prompt using an adapted visual language model, the second text prompt including a first object from the list of objects; and receive a set of coordinates of the first object in the image.

[0168] Aspect 44. The non-transitory computer-readable medium according to aspect 43, wherein the first text prompt includes a prompt for a list of objects in the image.

[0169] Aspect 45. The non-transitory computer-readable medium according to any one of aspects 43 to 44, wherein the second text prompt includes a prompt regarding the location of the first object.

[0170] Aspect 46. The non-transitory computer-readable medium according to any one of Aspects 43 to 45, wherein the visual language model and the adapted visual language model are the same machine language model.

[0171] Aspect 47. The non-transitory computer-readable medium according to any one of aspects 43 to 46, wherein the weights of the visual language model are frozen.

[0172] Aspect 48: An apparatus comprising one or more components for performing the operations described in any one or more of aspects 18 to 27.

[0173] Aspect 49: An apparatus comprising one or more components for performing the operations described in any one or more of aspects 28 to 32.< / obj>

Claims

1. An apparatus for adapting a pre-trained machine learning model, the apparatus comprising: One or more memories, the one or more memories being configured to store training images; and One or more processors, said one or more processors being coupled to said one or more memories and configured to: The training image is encoded into a first feature vector, the training image including a first object located at a first position; The second feature vector is generated using a set of weights based on a set of sinusoidal functions; The first feature vector is combined with the second feature vector to generate a combined feature vector. The combined feature vectors are processed using a visual language model to obtain the second position of the first object; and The weight set is adjusted based on a comparison between the first position and the second position.

2. The apparatus according to claim 1, wherein the set of sinusoidal functions includes cosine functions and sine functions.

3. The apparatus of claim 1, wherein the second position includes the text coordinates of the bounding box.

4. The apparatus of claim 1, wherein the combined feature vector and textual cues are processed using the visual language model to obtain the second position of the first object.

5. The apparatus of claim 4, wherein the text prompt includes a prompt indicating the location of the first object.

6. The apparatus of claim 1, wherein the one or more processors are configured to generate the training images.

7. The apparatus of claim 6, wherein, in order to generate the training image, the one or more processors are configured to insert the first object into a background image.

8. The apparatus of claim 7, wherein, in order to generate the training image, the one or more processors are configured to generate an image of the first object.

9. The apparatus of claim 8, wherein, in order to generate the image of the first object, the one or more processors are configured to submit object prompts to a text-to-image generation model.

10. The apparatus of claim 6, wherein the weights of the visual language model are frozen.

11. The apparatus of claim 1, wherein the one or more processors are configured to be trained on-device based on a machine learning system to adjust the set of weights.

12. The apparatus of claim 1, further comprising a camera configured to capture the training images.

13. The apparatus of claim 1, further comprising a display configured to display the output of the pre-trained machine learning model.

14. An apparatus for image processing, the apparatus comprising: One or more memories, the one or more memories being configured to store images; and One or more processors, said one or more processors being coupled to said one or more memories and configured to: The image and the first text prompt are processed using a visual language model to obtain a list of objects in the image; The image and the second text prompt are processed using an adapted visual language model, the second text prompt including a first object from the object list; as well as Receive the set of coordinates of the first object in the image.

15. The apparatus of claim 14, wherein the first text prompt includes a prompt for a list of objects in the image.

16. The apparatus of claim 14, wherein the second text prompt includes a prompt indicating the location of the first object.

17. The apparatus of claim 14, wherein the visual language model and the adapted visual language model are the same machine language model.

18. The apparatus of claim 14, wherein the weights of the visual language model are frozen.

19. The apparatus of claim 14, further comprising a camera configured to capture the image.

20. The apparatus of claim 14, wherein the set of coordinates of the first object includes the bounding box of the first object.

Images