System and method for adapting vision-language models using hypernetworks

Abstract

A computer-implemented method and system relate to the training of a vision-language model (VLM) that includes at least one image encoder and one text encoder. The VLM is trained using data pairs, where each data pair contains (i) image data of a digital image and (ii) text data describing that corresponding image data. The text encoder generates text embeds using the text data. A hypernetwork generates at least a subset of parameters for the image encoder using the text embeds. The image encoder generates image embeds using the image data, applying at least this subset of parameters. Data loss between the image embeds and the text embeds is minimized. The VLM and the hypernetwork are updated using this loss. The image encoder is relatively small and can be deployed on a resource-constrained device, such as an edge device.

Description

Technical field

[0001] The present disclosure relates generally to computer vision and in particular to the training and adaptation of vision-language models using hypernetworks. background

[0002] Self-supervised vision-language models (VLMs) trained with contrasting objectives perform better as their scale increases. Typically, the image encoders in such models are larger than the text encoders. The inference cost of the text encoder is often amortized using a predefined set of text embedding, but not with the image encoder. This poses a challenge for deploying large VLMs, especially in resource-constrained environments.

[0003] Furthermore, it is now common practice in deep learning to first pretrain a model on web-scale data and then adapt this model for a specific task using little or no additional data. Despite the widespread success of these models and their lack of dependence on large, labeled datasets, a significant drawback is that these models often have parameters on the order of billions—far larger than their supervised counterparts for a given task with the same level of accuracy.

[0004] The enormous size of image encoders in VLMs is a direct consequence of the scale of their pre-training datasets. These VLMs feature image encoders tasked with learning representations across an immensely large data domain. Small vision encoders, however, struggle to learn such a breadth of representations.

[0005] Although a variety of strategies exist for reducing the memory requirements or inference latency of these massive models, employing these strategies introduces some additional burdens. For example, these strategies are generally categorized into truncation, quantization, and distillation techniques. These techniques often involve first training a large model and then applying the chosen technique post-hoc. However, many of these methods may require specialized hardware support for actual memory and latency reduction. Brief description

[0006] The following is a summary of certain embodiments, which are described in detail below. The aspects described are presented merely to provide the reader with a brief summary of these certain embodiments, and the description of these aspects is not intended to limit the scope of protection of this disclosure. Indeed, this disclosure may include a variety of aspects that may not be expressly set forth below.

[0007] According to at least one aspect, a computer-implemented method involves training a machine learning model that includes an image encoder and a text encoder. The method involves receiving data pairs, where each data pair includes (i) image data of a digital image and (ii) text data describing that image data. The method involves generating, using the text encoder, text embeds based on the text data. The method involves generating, using a neural network, at least a subset of parameters for the image encoder using the text embeds. The method involves generating, using the image encoder, image embeds based on pixels of the image data while applying the subset of parameters. The method involves minimizing any loss between the image embeds and the text embeds.The process involves updating the machine learning model and the neural network using loss.

[0008] According to at least one aspect, a system includes at least one or more processors and one or more computer memories. The one or more computer memories are in data communication with the one or more processors. The one or more computer memories contain computer-readable data stored therein. The computer-readable data includes instructions which, when executed by one or more processors, cause the one or more processors to perform a procedure for training a machine learning model that includes an image encoder and a text encoder. The procedure involves receiving data pairs, where each data pair includes (i) image data of a digital image and (ii) text data describing that image data. The procedure involves generating, using the text encoder, text embeds based on the text data.The method involves generating, using a neural network, at least a subset of parameters for the image encoder using the text embeddings. The method involves generating, using the image encoder, image embeddings based on pixels of the image data while applying the subset of parameters. The method involves minimizing data loss between the image embeddings and the text embeddings. The method involves updating the machine learning model and the neural network using this data loss.

[0009] A computer-implemented method for training an image classifier that includes an image encoder. The image encoder is part of a machine learning model. The machine learning model includes the image encoder and a text encoder. The method involves receiving data pairs, where each data pair includes (i) image data comprising the pixels of a given digital image, and (ii) text data describing that image data. The method involves generating text embeds based on the text data using the text encoder. The method involves generating at least a subset of parameters for the image encoder using a neural network and the text embeds. The method involves generating image embeds based on the pixels of the image data using the image encoder while applying the subset of parameters.The procedure involves minimizing data loss between image and text embeddings. It involves updating the machine learning model and the neural network using this loss. The procedure involves receiving a set of class data for an image classification task. It involves generating class embeddings using the text encoder and the class data set. The procedure involves generating an updated set of parameters for the image encoder using the neural network. The image classifier includes the image encoder with the updated set of parameters. The image classifier uses the class embeddings to perform the image classification task.

[0010] These and other features, aspects, and advantages of the present invention are discussed in the following detailed description with reference to the accompanying drawings, in which identical symbols represent similar or identical parts. Furthermore, the drawings are not necessarily to scale, as some features may be exaggerated or minimized to show detail of certain components. Brief description of the characters Fig. Figure 1 is a diagram showing aspects of HyperCLIP according to an exemplary embodiment of this disclosure. Fig. Figure 2 is a diagram that shows an example of a hypernetwork architecture. Fig. 1 according to an exemplary embodiment of this disclosure. Fig. Figure 3 is a diagram of an example of a system with HyperCLIP according to an exemplary embodiment of this disclosure. Fig. Figure 4 is a diagram of an example process for generating a task-specific network using HyperCLIP according to an exemplary embodiment of this disclosure. Fig. Figure 5 illustrates an example of a smartphone with a task-specific network according to an exemplary embodiment of this disclosure. Fig. Figure 6 illustrates an example of an electrical device with a task-specific network according to an exemplary embodiment of this disclosure. Fig. Figure 7 illustrates an example of a control system with a task-specific network according to an exemplary embodiment of this disclosure. Fig. Figure 8 illustrates an example of a mobile machine technology that includes a control system with a task-specific network, according to an exemplary embodiment of this disclosure. Fig. Figure 9 illustrates an example of a manufacturing technology that includes a control system with a task-specific network, according to an exemplary embodiment of this disclosure. Fig. Figure 10 illustrates an example of a security technology that includes a control system with a task-specific network, according to an exemplary embodiment of this disclosure. Detailed description

[0011] The embodiments described herein, which have been shown and described by way of example, and many of their advantages are understood from the foregoing description, and it becomes apparent that various modifications to the shape, construction, and arrangement of the components can be made without departing from the disclosed subject matter or sacrificing one or more of its advantages. Indeed, the described forms of these embodiments are merely illustrative. These embodiments are susceptible to various modifications and alternative forms, and the following claims are intended to include and encompass such modifications and are not limited to the specific disclosed forms, but instead cover all modifications, equivalents, and alternatives that fall within the meaning and scope of protection of this disclosure.

[0012] Fig. Figure 1 is a diagram of an example system for training and adapting a vision-language model (VLM) using a hypernetwork 130. This system can be referred to as "HyperCLIP" 100, where "Hyper" refers to the hypernetwork and "CLIP" refers to contrastive speech-image pretraining (CLIP). The HyperCLIP 100 is a system that includes a process for pretraining or training a VLM to derive a small vision model (e.g., a small image encoder 120) suitable for use in resource-constrained systems (e.g., edge devices, etc.) without requiring multi-stage training procedures or any specialized hardware. The HyperCLIP 100 incorporates a novel architecture that improves performance over state-of-the-art baselines and can additionally be used in conjunction with a variety of model compression techniques for further memory or latency improvements.

[0013] At a high level, the HyperCLIP 100 incorporates a machine learning system that includes at least a VLM and a neural network. This machine learning system can be referred to as "the HyperCLIP model." The VLM includes at least the Text Encoder 110 and the Image Encoder 120. Meanwhile, the neural network includes at least the Hypernetwork 130. That is, as in Fig. As shown in Figure 1, the machine learning system (i.e., the HyperCLIP model) includes at least three main components: (i) a text encoder 110, (ii) an image encoder 120, and (iii) a hypernetwork 130.

[0014] The HyperCLIP model is pre-trained or trained using training data that includes data pairs. As an example, a data pair includes (i) image data and (ii) text data associated with that image data. The image data comprises pixels of a digital image. In digital imaging, a pixel is the smallest addressable element in a raster image or dot-matrix display device. In most digital displays, pixels are the smallest element that can be manipulated by software. Each pixel is a sample, or part, of a digital image. The intensity of each pixel is variable. Meanwhile, the text data includes a label associated with the corresponding image data. The text data can describe the image data. The label can include a keyword.

[0015] With reference to Fig. As a non-restrictive example, the training data consists of a batch of data pairs. These data pairs include text data (10) and image data (12). More precisely, this illustrates... Fig. 1. Three data pairs from at least part of a batch are provided as non-restrictive examples of training data. For example, the first data pair includes (i) text data 10A of "a photo of a dog" and (ii) corresponding image data 12A showing a dog. The second data pair includes (i) text data 10B of "a photo of a cat" and (ii) corresponding image data 12B showing a cat. The third data pair includes (i) text data 10C of "a photo of a truck" and (ii) corresponding image data 12C showing a truck. In these examples, for illustrative purposes, a keyword of "a photo of {object}" was used when generating each label, but the text data 10 does not require keywords and can contain any applicable image description associated with the image data 12.

[0016] With reference to Fig. The VLM includes the text encoder 110, which is designed to receive text data 10 and generate text embeddings 14 using the text data 10. In other words, the text encoder 110 is designed to receive text data 10 as input and generate one or more latent vectors (e.g., text embeddings 14) in an embedding space (e.g., the CLIP embedding space) as output. As an example, the text encoder 110 is based on a causal transformer architecture. Fig. 1. The text encoder 110 is trained from scratch to allow for greater freedom in determining the resulting contrastive embedding space. Alternatively, the text encoder 110 can incorporate a pre-trained text encoder (e.g., the CLIP text encoder) as needed.

[0017] In addition, the VLM includes the image encoder 120, which is designed to receive image data 12 and generate image embeddings 18 using at least (i) pixels of the image data 12 and (ii) an output (e.g., a subset of parameters 16) of the hypernetwork 130. In other words, the image encoder 120 is designed to receive image data 12 of at least one digital image as input and generate one or more latent vectors (e.g., image embeddings 18) in the same embedding space (e.g., the CLIP embedding space) as the text encoder 110. The image encoder 120 may have a similar functional form to the CLIP image encoder. However, the image encoder 120 differs from the CLIP image encoder. In this respect, the Image Encoder 120 is essentially smaller than the CLIP Image Encoder. For example, the total number of parameters of the Image Encoder 120 is considerably smaller than the total number of parameters of the CLIP Image Encoder.The Image Encoder 120 consumes fewer resources (e.g., memory, processing power, etc.) than the CLIP Image Encoder. The Image Encoder 120 is also more efficient and faster than the CLIP Image Encoder. The Image Encoder 120 has greater computational efficiency than the CLIP Image Encoder. Additionally, the Image Encoder 120 is smaller than the Text Encoder 110. For example, the total number of parameters for the Image Encoder 120 is smaller than the total number of parameters for the Text Encoder 110. Conversely, the CLIP Image Encoder is larger than the CLIP Text Encoder. Given its significantly smaller size and efficiencies, the Image Encoder 120 is designed to run on resource-constrained devices, whereas the CLIP Image Encoder may not run on those same resource-constrained devices, as the CLIP Image Encoder requires more resources than are available on those same resource-constrained devices.In this respect, the resource-constrained device may be limited in terms of memory, processing, latency, bandwidth, etc.

[0018] As discussed above, the Image Encoder 120 incorporates a small vision architecture. As a non-restrictive example, the small vision architecture can include EfficientNet (B0, B1, or B2), MobileNetV3 (M0 or M1), TinyNet (T0), EdgeNext (E0), or MobileViT (V0). Table 1 provides some details regarding these small vision architectures to emphasize their small scale. Specifically, Table 1 provides information regarding (i) “#PARAM (M)”, which specifies the total number of all parameters (represented on a scale of millions by “M” for mega) of the small vision architecture, (ii) “#ADAPT(K)”, which specifies the total number of parameters (represented on a scale of thousands by “K” for kilo) adapted by the HyperNetwork 130, and (iii) “TYPE ADAPT”, which specifies the type of adapted parameters.In Table 1, BN represents BatchNorm parameters, LN represents LayerNorm parameters, and GN represents GroupNorm parameters. As an illustrative example, if the EfficentNet B0 model is used as the small vision architecture for the Image Encoder 120, then the total number of all parameters of the Image Encoder 120 is 4.6 million, while the total number of customized parameters (i.e., BN parameters) of the Image Encoder 120 is 42,100. The choice of small vision architecture for the Image Encoder 120 largely depends on the target architecture of a technical system at development time. TABLE 1 MODELL B0 B1 B2 M0 M1 T0 E0 V0 # PARAM(M) 4,6 7,2 8,4 4,9 2,0 1,7 7,6 4,7 # ADAPT (K) 42,1 62,1 67,6 24,4 12,1 17,1 8,8 15,5 TYPEADAPT BN BN BN BN BN BN LN BN & GN

[0019] Again with reference to Fig. 1. HyperCLIP 100 incorporates a new component during the development process of the small image encoder 120. This new component is the hypernetwork 130. The hypernetwork 130 is designed to map the text embeddings 14 to specific parameters of the image encoder 120 itself. In this respect, the hypernetwork 130 automatically generates at least a subset of parameters 16 (e.g., relevant parameters) of the image encoder 120 based on the specific upcoming task. Specifically, the hypernetwork 130 takes as input the set of text embeddings 14 generated by the text encoder 110 and generates as output a subset of parameters 16 of the image encoder 120. The image encoder 120 is designed to apply at least this subset of parameters 16 when the image embeddings 18 are generated using the pixels of the image data 12 of the digital images.In this respect, the insight here is that a suitably large hypernetwork 130 can contain the logic about how to “specialize” the image encoder 120 for a given task, precisely the task specified for embedding images that are assumed to be associated with one of the provided text embeddings.

[0020] Fig. Figure 2 shows aspects of an example of the HyperCLIP 100 hypernetwork 130 according to an exemplary embodiment. As an overview, the Hypernetwork 130 takes as input a set of text embeds 14 and outputs at least a subset of parameters 16 for the target image encoder 120. For this purpose, the Hypernetwork 130 includes in the Fig. In the example shown, at least a linear layer 132, a transformer model 134, a bottleneck layer 136, and an average pool and linear layer 138 are included. More precisely, the linear layer 132 is designed to (i) receive a batch of text embeddings 14 as input vectors with input dimensions and (ii) generate the text embeddings 14 into intermediate vectors with predetermined dimensions, which can be called projected text embeddings and are compatible with the requirements of the transformer model 134. As a non-restrictive example, the linear layer 132 includes an input projection layer with learnable weights FF. inputThe Transformer Model 134 comprises a deep learning architecture with a plurality of Transformer layers (i.e., self-attention layers) designed to (i) receive the projected text embeddings as input, (ii) "mix" the input information and learn to distinguish classes and concepts represented by the projected text embeddings, and (iii) generate initial intermediate vectors of parameters as output. As a non-restrictive example, the Transformer Model 134 is a Transformer encoder comprising a twelve-layer Transformer Model 134 with a width of 768, 8 heads, a T-feed-forward dimension of 2560 with GELU activation, no masking, and a dropout of 0.1.The bottleneck layer 136 is designed to convert the first intermediate vectors of parameters into second intermediate vectors of parameters, where one dimension of the second intermediate vector of parameters is smaller than one dimension of the first intermediate vector of parameters. In other words, the bottleneck layer 136 produces an output that is a compressed representation of its input. The average pool and linear layer 138 is designed to (i) receive the second intermediate vectors of parameters, (ii) generate third intermediate vectors of average values ​​of parameters associated with an entire batch of text embeddings, (iii) transform third intermediate vectors of average values ​​with certain dimensions into output vectors with predetermined output dimensions, and (iv) output at least the output vectors that represent at least a subset of parameters 16 (e.g.,The subset of parameters for the image encoder 120 includes normalization parameters. In this case, the subset of parameters includes 16 normalization parameters. The normalization parameters include scaling and bias parameters. The subset of parameters forms a single set of normalization parameters for the batch of text embeddings. As a non-restrictive example, the average pool and linear layer 138 includes a layer normalization LN and an output feed-forward layer FF. output The output dimension of the output feed-forward layer FF output The number of parameters that are adjusted for the image encoder is 16.

[0021] As discussed above, the hypernetwork 130 with this configuration is designed to process text embeddings 14 using the transformer model 134 and at least a subset of parameters 16 (e.g., normalization parameters) directly. This setting leads to some natural limitations and invariances that are desirable in the hypernetwork 130 itself, as well as important considerations about which parameters are generated. For example, with respect to the hypernetwork setting, the hypernetwork 130 should accept any number of text embeds 14 as input. The hypernetwork 130 should generate an appropriate image encoder 130 not only for a fixed batch size of potential keywords, but indeed for any number of keywords (up to a reasonable limit of the size constraints).In addition, the hypernetwork 130 should be invariant with respect to the order of these text embeddings with respect to the hypernetwork setting: the "order" of the keywords provided to the hypernetwork 130 is completely random and should have no effect on the target image encoder 120. Fortunately, the transformer model 134 (with variably sized collections of inputs and without causal masking or positional coding) satisfies both of these requirements. Thus, the hypernetwork 130 comprises a non-causal transformer model 134, where each individual keyword embedding serves as a single "token" that is fed into the transformer model, which is used to generate the final parameters of the image encoder 120.Alternatively, Hypernetwork 130 can also use global average pooling across the last layer of embeddings in Hypernetwork 130, although this makes little difference to performance in practice. The resulting Hypernetwork 130 is designed to take all of the input keywords and output a single set of image encoder parameters, which creates an image encoder 120 capable of distinguishing maximally between images that match all such keywords.

[0022] In Fig. In the second embodiment, HyperNetwork 130 adopts the approach of modifying only the normalization (e.g., BN, LN, GN), bias, and scaling parameters of the target image encoder 120. In alternative embodiments, HyperNetwork 130 is designed to output all parameters of the image encoder 120. More precisely, small image encoders 120 typically have tens of thousands of such parameters, making them a valuable target for HyperNetwork 130 in that they are still known to provide a very powerful control surface of the target model (i.e., the image encoder 120) while being relatively small in number. HyperCLIP 100 also trains the remaining parameters (i.e.,, convolution filters and multilayer perceptron (MLP) weights) of the image encoder 120, but the HyperCLIP 100 does this in a way that is shared across all of the different keywords within the training: that is, these non-BN / LN parameters are shared across all different batches of training, while only the BN / LN parameters are the subset of parameters 16 that are fitted according to the output of the hypernetwork 130.

[0023] Again with reference to Fig. 1. As an overview, the HyperCLIP 100 trains the text encoder 110, the image encoder 120, and the hypernetwork 130 simultaneously using contrastive loss, SigLip-based loss, or an applicable loss function. The loss function involves calculating a scalar product 20 between the text embeddings 14 and the image embeddings 18 to calculate their similarity. It should be noted that at test time, only the small image encoder 120, which is actually generated by the hypernetwork 130 based on the desired set of class data 22 (e.g., class keywords), is used, as shown in Fig. 4 shown and discussed. In other words, the image encoder 120, generated by the HyperCLIP 100, can be directly applied to efficient test time classification without the need for a separate distillation phase to “shrink” the network to a smaller target architecture.

[0024] More formally, the HyperCLIP 100 can be described as a predecessor using the following terms. For a given image encoder (e.g., Image Encoder 120), F:ℝbatch×img→ℝbatch×emb; Text encoder (e.g. Text Encoder 110), G:ℝbatch×ctx→ℝbatch×emb; and hypernetwork H:ℝbatch×emb→ℝmdim, The training goal is siglip : ℝ batch×emb → ℝ batch×emb × ℝ batch×emb = ℝ batch , and the zero-shot inference matrix is ​​sim : ℝ batch×emb × ℝ classes×emb → ℝ batch×classes Furthermore, the image encoder has 120 parameters Θ = {Θ1 ... Θ L} on, where Θ lParameters of each layer. Furthermore, L, batch, classes, ctx, emb, img, mdim ∈ ℝ, where L represents the number of layers of the image encoder 120, batch represents the number of data pairs in a batch, classes represents the number of classes, ctx represents the dimensionality of the text input, emb represents the dimensionality of each embedding, img represents the dimensionality of the image input, and mdim represents the number of parameters output by the hypernetwork 130 (i.e., the number of parameters of the image encoder 120 that are modified by the hypernetwork 130).

[0025] With reference to the formal designations described above, HyperCLIP includes 100 training and inference steps, as described below. Regarding image embedding... X=F(Images;Θ) and the text embedding Y=G(labels) The hypernetwork takes 130, H(Y;Φ), The text embeddings Y are taken as input and dynamically generate at least a subset of parameters 16 for the image encoder 120. Here, Φ represents the weights of the hypernetwork 130. The HyperCLIP 100 defines Θ' = {γ, β}, which specifically refers to the normalization parameters generated by the hypernetwork 130. The loss function is defined similarly to SigLIP loss, but with dynamically generated normalization parameters. X'=F(Images;Θfixed,Θ') |b|=(2∗I)−1 hyperclip(X',Y;η,ς)=−logsigmoidbatch(|b|∗(η∗(X'⊙Y)+ζ))

[0026] In equation 1, Θ represents fixedThe fixed parameters of the image encoder 120, while Θ' represents the normalization parameters (i.e., the subset of parameters 16) generated by the hypernetwork 130. The image embedding X' is obtained using both fixed and dynamic parameters in the image encoder 120. The "fixed" parameters continue to be updated during training. In Equation 2, |b|=(2∗I)−1 as a matrix of ones on the diagonals and ones elsewhere. In Equation 3, HyperCLIP defines a similarity measure sim(X', Y) = X' ⊙ Y between a given image and text embedding, where ⊙ is the matrix product. This measure enables an inference rule, such as pred = argmax sim(X', Y), which is used to predict a text label for each class. Furthermore, the process during training involves batch optimization of loss, as expressed in Equation 4. Additionally, η, ζ ∈ ℝ are parameters in Equations 3 and 4. Loss=min∑batchhyperclip(X'Y;η,ζ)

[0027] Finally, the process can involve fine-tuning a linear layer 402 (i.e., linear probe) of the image encoder 120, its weights being initialized with Y by Equation 5, where Y* ∈ R batch Evaluation labels are assigned to each digital image. Probe=minimize∑batch−logsoftmaxbatch(X')⊙Y*

[0028] For zero-shot classification, X' is explicitly conditioned on Y before the Argmax using the hypernetwork 130, as expressed in equation 6. sim(X',Y)=X'⊙Y pred=argmaxclasses sim(X',Y)

[0029] During training, HyperCLIP 100 freezes the normalization parameters (i.e., the subset of parameters 16), keeps the scaling parameters γ positive by applying the exponential function, and uses the running average estimation of population statistics. The image embeddings 18 are only obtained after the normalization parameters (or the subset of parameters 16) of the image encoder 120 have been modified by the hypernetwork 130 during the forward pass. During the reverse pass, the text encoder 110, the remaining parameters of the image encoder 120, and the hypernetwork 130 are updated using the gradient of the SigLIP loss, which is calculated using Y and X. Furthermore, HyperCLIP 100 is designed to obtain the desired keywords and use them to fix the parameters of the associated image encoder 120 before inference begins.Since the HyperCLIP 100 does not modify the image encoder 120 at inference time or add any parameters to it, the costs remain unchanged relative to a baseline model.

[0030] Fig. Figure 3 is a diagram of an example of a System 300 with a HyperCLIP 100 according to an exemplary embodiment of this disclosure. The System 300 includes at least one processing system 302. The processing system 302 includes at least one electronic processor, a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor, a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), any suitable processing technology, or any number and combination thereof. The processing system 302 is functional to provide the functionality described herein.

[0031] System 300 includes at least one storage system 304, which is operationally connected to the processing system 302. The storage system 304 is in data communication with the processing system 302. In an exemplary embodiment, the storage system 304 includes at least one non-volatile, computer-readable medium designed to store various data and provide access to it, enabling at least the processing system 302 to perform the operations and functionalities disclosed herein. In an exemplary embodiment, the storage system 304 comprises a single device or a plurality of devices. The storage system 304 may include an electrical, electronic, magnetic, optical, semiconductor, electromagnetic, or any suitable storage technology that is operable with System 300.For example, in an exemplary embodiment, the storage system 304 may include random access memory (RAM), read-only memory (ROM), flash memory, a hard disk, a memory card, an optical storage device, a magnetic storage device, a memory module, any suitable type of storage device, or any combination thereof.

[0032] Storage system 304 contains at least the HyperCLIP 100, machine learning (ML) data 306, and other relevant data 308 stored on it. Storage system 304 contains computer-readable data designed, when executed by processing system 302, to implement a pre-training or training process of the HyperCLIP 100 to perform the functions as described in at least... Fig. 1, Fig. 2 and Fig. 4 described in the disclosure. The computer-readable data may include instructions, code, routines, various associated data, any software technology, or any number and combination thereof. In particular, HyperCLIP 100 comprises a machine learning system that includes (i) a machine learning model (e.g., a VLM) comprising at least one text encoder 110 and one image encoder 120, and (ii) a neural network comprising at least one hypernetwork 130. Furthermore, the ML data 306 includes various training data, various loss data, various weight data, and / or parameter data, as well as any associated machine learning data that enables the system 300 to perform the functions disclosed in this disclosure. The training data includes various pairs of text data and image data, wherein each text data of a data pair describes corresponding image data of that data pair.Meanwhile, the other relevant data 308 provide different data (e.g., operating system, etc.) that enable the system 300 to perform the functions discussed herein.

[0033] In an exemplary embodiment, as in Fig. As shown in Figure 3, the system 300 is designed to include at least one sensor system 310. The sensor system 310 includes one or more sensors. For example, the sensor system 310 includes an image sensor or a camera. The sensor system can also include a radar sensor, a LiDAR sensor (light detection and distance measurement), a thermal sensor, an ultrasonic sensor, an infrared sensor, a motion sensor, an audio sensor, an inertial measurement unit (IMU), any suitable sensor, or any combination thereof. The sensor system 310 is capable of communicating with one or more other components (e.g., the processing system 302 and the storage system 304) of the system 300. More specifically, the processing system 302 is designed, for example, to receive the sensor data directly or indirectly from at least the image sensor. The sensor data can also be taken from one or more sensors of the sensor system 310.After receiving the sensor data, the processing system 302 is designed to process this sensor data (e.g. digital image) in conjunction with the HyperCLIP 100 and the ML data 306.

[0034] In addition, the system includes 300 other components that contribute to HyperCLIP 100. For example, in Fig. As shown in Figure 3, the storage system 304 is also designed to store other relevant data 308 relating to the operation of the system 300 with respect to one or more components (e.g., sensor system 310, an input / output (I / O) system 312, and other functional modules 314). Additionally, the I / O system 312 includes an I / O interface and may include one or more devices (e.g., display device, keyboard device, speaker device, etc.). Furthermore, the system 300 includes other functional modules 314, such as any suitable hardware technology, software technology, or combination thereof, that support or contribute to the functionality of the system 300. For example, the other functional modules 314 include communication technology that enables components of the system 300 to communicate with each other, as described herein.The communication technology can enable System 300 to communicate with other network devices (not shown) via a communication network. At least with the one in the example of... Fig. In the configuration discussed in section 3, the System 300 is operational for the HyperCLIP 100 to perform the process and functions as discussed in this disclosure.

[0035] Fig. Figure 4 is a diagram illustrating aspects of an example process for generating a task-specific network 400 according to an exemplary embodiment. This process can be carried out by one or more processors of the processing system 302 ( Fig. 3) are carried out. Furthermore, this process uses the trained HyperCLIP model (e.g., trained text encoder 110, trained image encoder 120, and trained hypernetwork 130) to generate the task-specific network 400. That is, this process takes place after the pre-training or training process of Fig. 1 instead. Furthermore, in this specific example, the process involves generating a task-specific network 400 for image classification based on the set of class data 22. In particular, in Fig. 4. The task-specific network 400 is an image classifier that includes at least the trained image encoder 120, the linear layer 402, and logits computations. Alternatively, the trained image encoder 120 can be part of a task-specific network 400 that is further trained to perform another specific task, such as data set shifting, linear probing tasks, image retrieval recall, or any applicable computer vision task.

[0036] With reference to Fig. 4 As a non-restrictive example, the process for a specific image classification task involves receiving or obtaining a set of class data 22. The class data 22 may include a class name, a class description, or any similar descriptive text data. With reference to Fig. As a non-restrictive example, the set of class data 22 includes at least "Pretzel", "Muffins", "Pizza" and other food labels / descriptions / names. The set of class data 22 is passed to the trained text encoder 110.

[0037] The trained text encoder 110 is configured to receive the set of class data 22. The trained text encoder 110 is configured to generate a set of class embeddings 24 using the set of class data 22. The set of class embeddings 24 is transferred to (i) the trained hypernetwork 130 and (ii) the linear layer 402. In the first transfer example, the trained hypernetwork 130 is configured to receive the set of class embeddings 24 and generate at least one updated subset of parameters (e.g., normalization parameters 26) for the trained image encoder 120. The image encoder 120 is updated using at least this updated subset of parameters. Furthermore, in the second transfer example, the linear layer 402 is configured to receive the class embeddings 24 from the trained text encoder 110. The class embeddings 24 serve as weights for the linear layer 402.After performing these updates, the task-specific network 400 can be used and / or deployed as an image classifier.

[0038] For an image classification task, the trained image encoder 120 is configured to receive at least one digital image 28. The trained image encoder 120 is configured to generate image embeddings 30 using pixels from image data of at least one digital image 28, while at least the updated subset of parameters (e.g., normalization parameters 26) is applied and / or used by the trained image encoder 120. The linear layer 402 receives the image embeddings 30. The linear layer 402 generates a result by transforming the image embeddings 30, while the class embeddings 24 are used as weights. Next, the logits 32 are calculated based on the result. The logits 32 form the likelihoods over the set of class data 22.For class prediction, the process involves taking the class data 22 that are associated with the highest probability or likelihood, taken from the logits 32. In this case, the task-specific network 400 is designed to (i) determine that "Pizza" is the class data with the highest probability or likelihood, using the logits 32, and (ii) generate output data 34 of "Pizza" as the class data that classifies the digital image 28.

[0039] Fig. Figure 5 illustrates an example of the deployment and use of the task-specific network 400 on a resource-constrained computing device according to an exemplary embodiment. As mentioned earlier, the task-specific network 400 is relatively small and therefore deployable and usable on resource-constrained devices. For example, the resource-constrained device could be a kiosk machine. The resource-constrained device could be an edge device. The resource-constrained device could be an Internet of Things (IoT) device.

[0040] In Fig. 5 The task-specific network 400 is used as a non-restrictive example on a mobile device, such as a smartphone 500. The smartphone 500 includes at least one camera 510 designed to capture and produce digital images (e.g., digital image 28) and / or digital video. The smartphone 500 also includes at least one processing device (not shown) and at least one memory 520. The memory 520 contains computer-readable data with instructions stored on it. The computer-readable data, which is executable by at least one processor or at least one processing device, includes at least one computer vision application 530 and a task-specific model 400. The smartphone 500 is designed to use the task-specific network 400 to produce output data 34 (“Pizza”) that classifies the digital image 28.In this example, the computer vision application 530, together with the task-specific network 400, can be used to help a user classify, identify, describe, and / or tag digital images captured, received, or obtained by the smartphone 500. The smartphone 500, using the computer vision application 530, can be configured to display the output data (e.g., pizza) of the task-specific network 400, along with other information (e.g., digital image 28, etc.) relating to this output data 34.

[0041] Fig. Figure 6 illustrates an example of the deployment and use of the task-specific network 400 in a resource-constrained environment according to an exemplary embodiment. As mentioned previously, the task-specific network 400 is relatively small and can therefore be deployed and used on resource-constrained devices. For example, the resource-constrained device could be an electrical appliance.

[0042] In Fig. 6 The task-specific network 400 is used as a non-restrictive example on a household appliance, such as an oven 600. The oven 600 can be a smart oven. The oven 600 includes at least one camera 610 designed to capture and produce digital images (e.g., digital image 28) and / or digital video. The oven 600 also includes at least one processing device (not shown) and at least one memory 620. The memory 620 contains computer-readable data with instructions stored on it. The computer-readable data, which is executable by at least one processor or at least one processing device, includes at least one computer vision application 630 and a task-specific model 400. The computer vision application 630 is an application program that uses the output data 34 (e.g., pizza) of the task-specific network and presents this information to the user.The oven 600 can include a display device for showing the output data (e.g., pizza) along with other information (e.g., recommended oven / cooking settings) regarding this output data 34. The display device can also show the input (e.g., digital image 28) of the task-specific network 400. For example, in this non-restrictive example, the oven 600 receives at least one digital image 28 of a pizza and furthermore generates output data 34 (“pizza”) that classifies the digital image 28. In this case, the computer vision application 630 and / or the task-specific network 400 can be used to assist a user in classifying one or more items 630 for cooking in the oven 600 using the digital image 28. After the classification task has been performed, the oven 600 and / or the computer vision application 630 are designed to display the cooking settings (e.g., baking mode, cooking time, temperature, etc.).) to automatically recommend and / or set 640 for this matter.

[0043] Fig. Figure 7 illustrates another example of a system 700 with a relatively small task-specific network 400 according to an exemplary embodiment. In this example, the system 700 includes at least one sensor system 710, a control system 720, and an actuator system 730. The system 700 is designed such that the control system 720 controls the actuator system 730 based on sensor data from the sensor system 710. In particular, the sensor system 710 includes one or more sensors and / or corresponding devices for generating sensor data. For example, the sensor system 710 includes at least one image sensor, a radar sensor, a LiDAR sensor, a thermal sensor, an ultrasonic sensor, an infrared sensor, a motion sensor, a satellite-based navigation sensor (e.g., a global positioning system (GPS) sensor), an optical sensor, an audio sensor, any suitable sensor, or any combination thereof.After receiving detections from its environment, the sensor system 710 is capable of communicating with the control system 720 via an I / O system 760 and / or other functional modules 770 that incorporate communication technology. The control system 720 is designed to receive sensor data directly or indirectly from one or more sensors of the sensor system 710. In this respect, the sensor data can include sensor data from a single sensor or sensor fusion data from multiple sensors. After receiving the input, which includes at least sensor data, the control system 720 is capable of processing the sensor data using a processing system 740 to ensure that the sensor data is in a suitable form (e.g., digital images) for the task-specific network 400.

[0044] The 740 processing system includes at least one processor. For example, the 740 processing system includes an electronic processor, a CPU, a GPU, a microprocessor, an FPGA, an ASIC, processing circuits, any suitable processing technology, or any combination thereof. After processing at least this sensor data (e.g., a digital image), the 740 processing system is capable of generating output data (e.g., classification from the task-specific network 400) based on communications with the 750 storage system. Additionally, the 740 processing system is capable of providing actuator control data to the 730 actuator system based on the output data.

[0045] The Storage System 750 is a computer or electronic storage system designed to store and provide access to various types of data in order to enable at least the operations and functionalities disclosed herein. The Storage System 750 comprises a single device or a plurality of devices. The Storage System 750 includes an electrical, electronic, magnetic, optical, semiconductor, electromagnetic, or any suitable storage technology, or any combination thereof. For example, the Storage System 750 may include RAM, ROM, flash memory, a hard disk, a memory card, an optical storage device, a magnetic storage device, a memory module, any suitable type of storage device, or any combination thereof.

[0046] The storage system 750 includes at least one computer vision application 780, a task-specific network 400, and other relevant data 790, each designed to be executed and / or implemented by the processing system 740. The computer vision application 780 is designed to provide a computer vision technology application program using the output of the task-specific network 400. The storage system 750 includes computer-readable data which, when executed by the processing system 740, is designed to execute the computer vision application 480 and utilize the task-specific network 400 to perform a specific task (e.g., image classification tasks, data set shifting tasks, linear probing tasks, image retrieval, etc.).The computer-readable data can include instructions, code, routines, various related data, any software technology, or any number and combination thereof.

[0047] As mentioned previously, the task-specific network 400 includes at least the trained image encoder 120 and is further configured to perform a specific task. For example, in Fig. 4, Fig. 5 and Fig. 6. The task-specific network 400 is set up as an image classifier using the set of class data 22 (e.g., food descriptions) to classify digital images according to this set of class data 22. Additionally, it includes Fig. 8, Fig. 9 and Fig. 10 different task-specific networks (400) configured as image classifiers. In particular, each of Fig. 8. Fig. 9 and Fig. 10 a control system 720 with a task-specific network 400, which uses a similar process to that of Fig. 4 is set up for an image classifier, but with a different set of class data 22 regarding its target application. For example, Fig. 8 include a task-specific network 400 that is set up as an image classifier with a set of class data 22 relating to driving scene objects (e.g., road signs, motorcycles, vehicles, pedestrians, bicycles, etc.) encountered while controlling a vehicle. In contrast, Fig. 9 includes a task-specific network 400, which is set up as an image classifier with a set of class data 22 relating to states of a manufactured product 902. As yet another example, Fig. 10. A task-specific network 400 is included, configured as an image classifier with a set of class data 22 relating to security detections (e.g., Person 1, Person 2, Dog, Cat, Bird, etc.) that may be encountered around a door 1002. In general, in these different examples, the task-specific network 400 refers to a vision model that includes at least the trained image encoder 120 and is configured to perform a specific task. In these examples, the task-specific network 400 is designed to perform an image classification task, but the task-specific network can be designed and configured to perform a different task (e.g., data set shifting tasks, linear probing tasks, image retrieval, etc.) for a target computer vision application.

[0048] Furthermore, as in Fig. As shown in Figure 7, the system 700 includes other components that contribute to the operation of the control system 720 with respect to the sensor system 710 and the actuator system 730. For example, in Fig. As shown in Figure 7, the storage system 750 is also designed to store other relevant data 790 relating to the operation of the system 700 with respect to one or more components (e.g., sensor system 710, actuator system 730, etc.). Furthermore, as Fig. As shown in Figure 7, the control system 720 includes the I / O system 760, which provides one or more interfaces for one or more I / O devices relating to the system 700. For example, the I / O system 760 provides at least one interface to the sensor system 710 and at least one interface to the actuator system 730. Furthermore, the control system 720 is designed to provide other functional modules 770, such as any suitable hardware technology, software technology, or any combination thereof, that support and / or contribute to the functionality of the system 700. For example, the other functional modules 770 include an operating system and communication technology that enables components of the system 700 to communicate with each other, as described herein. With at least the one shown in the example of Fig. The System 700 is applicable to various technologies in the configuration discussed in section 7.

[0049] Fig. Figure 8 is a diagram of the system 700 with respect to a mobile machine technology 800 according to an exemplary embodiment. The mobile machine technology 800 can be any mobile machine that includes at least a sensor system 720, a control system 710, and an actuator system 730. As a non-limiting example, the mobile machine technology 800 includes in Fig. 8 at least a partially autonomous vehicle. The mobile machine technology 800 is at least a partially autonomous vehicle that includes the sensor system 710. One or more of the sensors can be integrated with respect to the vehicle.

[0050] The 720 control system is designed to receive image data (e.g., digital images) based on sensor data or sensor fusion data from the 710 sensor system. The 720 control system is designed to detect objects in the vicinity of the vehicle based on this sensor data. The 720 control system is designed to provide input data to the 780 computer vision application and the 400 task-specific network. The 400 task-specific network is designed to classify the digital images received from the 710 sensor system with respect to autonomous driving. For example, as a non-restrictive example, the 400 task-specific network is designed to classify a digital image as belonging with a highest likelihood to the "stop sign" class. The 720 control system is designed to generate actuator control data for a braking operation in response to the "stop sign" classification of the object.In this case, the actuator system 730 is designed to stop the vehicle after receiving the actuator control data. In this respect, the actuator system 730 can include a braking system, a drive system, an engine, a powertrain, a steering system, and / or any applicable actuation system of the vehicle. The actuator system 730 is designed to control the vehicle so that, using the computer vision application 780, the vehicle follows traffic rules and avoids collisions based on classifications provided by the task-specific network 400.

[0051] Additionally, as another non-restrictive example, mobile machine technology 800 includes at least one partially autonomous robot. The robot can be an edge device. As a non-restrictive example, mobile machine technology can be a robotic vacuum cleaner, a robotic lawnmower, a cleaning robot, etc. As another non-restrictive example, mobile machine technology can be a drone. For example, the robot is designed to perform one or more functions such as flying, driving, walking, maneuvering, etc. The robot can be at least a partially autonomous lawnmower or a partially autonomous cleaning robot. In this respect, the actuator system 730 is designed to control, drive, steer, or stop the robot in such a way that the robot avoids collisions based on image classifications provided by the task-specific network 400.

[0052] Furthermore, as another non-restrictive example, the mobile machine technology 800 includes at least a partially autonomous robot in the form of a garden robot. In this example, the control system 720 is designed to provide input images to the task-specific network 400 based on sensor data. The task-specific network 400 is designed to classify these input images to identify the condition of plants in the environment and / or the species of plants present. The control system 720 is further designed to generate actuator control data based on these classifications (e.g., plant condition or identified plant species), so that the actuator system 730 is designed to deliver an appropriate amount of water, garden chemicals, and / or treatments.

[0053] Fig. Figure 9 is a diagram of the system 700 with respect to manufacturing technology 900 according to an exemplary embodiment. As a non-limiting example, the manufacturing technology 900 includes a punching machine, a cutter, a deep-hole drill, or any suitable type of manufacturing machine. Fig. 9. The sensor system 710 includes at least one image sensor or optical sensor. The control system 720 is designed to receive image data from the sensor system 710. The task-specific network 400 is designed to classify each digital image that shows a state of a manufactured product 902. For example, the control system 720 can classify a current state of the manufactured product 902 from various states in the manufacturing process. The control system 720 is designed to determine or select actuator control data in response to the classification of the current state of the manufactured product 902 based on properties detected by the sensor system 710. For example, as a non-restrictive example, the actuator control data can cause the control system 720 to trigger the next manufacturing step 904 of the manufacturing process based on the classified state of the manufactured product 902.

[0054] Fig. Figure 10 is a diagram of system 700 with respect to security technology 1000 according to an exemplary embodiment. As a non-limiting example, the security technology 1000 includes at least a monitoring system, an access control system, an observation system, or any suitable type of security device. For example, Fig.10 refers, as an example, to security technology 1000, which is designed to physically control a locked and an unlocked state of door 1002. The sensor system 710 includes at least one image sensor designed to capture digital images and / or digital video. The control system 720 is designed to receive the digital images and / or digital video from the sensor system 710. The control system 720 is designed to provide a digital image to the task-specific network 400. For example, the task-specific network 400 can classify objects that are typically located around a particular door. For example, the task-specific network 400 can classify image data from a digital image as including a face image belonging to person 1, person 2, ..., or person N, where N represents an integer.Additionally or alternatively, the task-specific network 400 can classify animals such as dog, cat, fox, deer, etc. The control system 720 is designed to generate actuator control data in response to the classification output by the task-specific network 400. For example, as a non-restrictive example, the actuator control data can cause the control system 720 to lock or unlock door 1002 when the task-specific network 400 identifies the input image as belonging to Person 3. Additionally or alternatively, as another non-restrictive example, the actuator control data can cause the control system 720 to display the input data (e.g., digital image or digital video) on the display device 1004 and / or the output data (e.g., Person 3) and / or other relevant data.The actuator control data can also cause the control system 720 to transmit this specific digital image and / or digital video, along with the corresponding output data from the task-specific network 400, to the relevant authorities.

[0055] As described in this disclosure, HyperCLIP 100 incorporates a number of advantageous features and benefits. For example, HyperCLIP 100 incorporates a novel architecture designed to enhance VLMs by dynamically adapting the image encoder 120 using a hypernetwork 130. More specifically, HyperCLIP 100 incorporates at least one novel hypernetwork 130 that takes text embeddings 14 from the text encoder 110 of a VLM and outputs at least the subset of parameters 16 (e.g., weights) of the image encoder 120 of the VLM. In this way, the hypernetwork 130 learns the model weights necessary to represent an image as a function of text associated with that image. This hypernetwork 130 is trained together with a text encoder 110 of the VLM and an image encoder 120 of the VLM and is compatible with any type of contrastive pretraining.

[0056] HyperCLIP 100 incorporates a method and system that enables the use of a much smaller Image Encoder 120, resulting in inherent compression—that is, fewer model parameters and faster inference. In this respect, HyperCLIP 100 addresses the challenge of deploying large VLMs in resource-constrained environments (e.g., memory-limited environments) by generating a significantly smaller, task-specific Image Encoder (e.g., Image Encoder 120) that maintains high performance. HyperCLIP 100 is advantageous when deploying an Image Encoder 120 at a smaller scale or with a reduced size. Furthermore, the performance of these small vision models can be improved by several percentage points across a range of tasks when their weights are adjusted using HyperCLIP 100.In some cases, a small vision model trained using the HyperCLIP 100 is able to outperform a larger, unadapted vision model.

[0057] Furthermore, by conditioning the image encoder parameters on the text embeddings, the HyperCLIP 100 achieves consistent and significant improvements in zero-shot accuracy, robustness against distribution shifts, and fairness metrics without requiring extensive post-hoc optimization or specialized hardware. Moreover, the HyperCLIP's ability to generate efficient and high-performance VLMs has implications for the democratization of computer vision models, enabling their use on resource-constrained devices and in diverse situations. Additionally, its improved fairness metrics and robustness against distribution shifts can help mitigate biases and enhance the inclusivity of computer vision models across various applications.

[0058] Additionally, the HyperCLIP 100 incorporates an architecture for learning transferable vision models that are resource-efficient and exhibit the same performance as their larger, non-hypernetwork-enhanced counterparts. The HyperCLIP 100 dynamically adjusts the vision model's weights during training, eliminating the need for post-hoc optimization. Furthermore, using the HyperCLIP 100 to adjust only the normalization layers of several widely used small vision models is sufficient to improve their performance against standard zero-shot classification benchmarks. Additionally, the HyperCLIP 100 has been demonstrated to improve baseline-relative performance in several distribution shift and fairness tasks.

[0059] Furthermore, instead of fixing vision encoders to accommodate all possible image labels, the HyperCLIP 100 offers a novel strategy: a method and system designed to adaptively precondition the image encoder 120 based on each specific text input. By intelligently adjusting the weights of the image encoder 120, this allows for the use of a much smaller image encoder vision network (e.g., a task-specific network 400) that is automatically specialized for a given task.

[0060] Additionally, the HyperCLIP 100 is advantageously designed to directly train a VLM, completely bypassing an explicit distillation process, and instead generates an image encoder 120 already optimized for use with a specific classification problem. To achieve this, the HyperCLIP 100 utilizes a hypernetwork 130 that directly generates a specialized image encoder 120 for a subset of textual keywords. The HyperCLIP 100 is designed to deploy a classifier with the specialized image encoder 120 on a small embedded device, an edge device, or a small-scale technology.

[0061] Furthermore, the above description is intended to be illustrative and not limiting, and is provided in the context of a specific application and its requirements. Those skilled in the field will recognize from the foregoing description that the present invention can be implemented in a variety of forms, and that the different embodiments can be implemented alone or in combination.Therefore, although the embodiments of the present invention have been described in conjunction with certain examples thereof, the general principles defined herein can be applied to other embodiments and applications without deviating from the meaning and scope of protection of the described embodiments, and the true scope of protection of the embodiments and / or methods of the present invention is not limited to the embodiments shown and described, since various modifications will be apparent to a person skilled in the art upon examination of the drawings, the patent specification, and the following claims. Additionally or alternatively, components and functionality may be separate or combined differently than in the manner of the various described embodiments and may be described using different terminology.These and other variations, modifications, additions and improvements may fall within the scope of protection of the disclosure as defined in the following claims.

Claims

[1] Computer-implemented method for training a machine learning model which includes an image encoder and a text encoder, wherein the computer-implemented method comprises: Receiving data pairs containing image data and text data, where the text data describes the corresponding image data of a digital image; Generating text embeddings based on the text data using the text encoder; Generating, using a neural network, at least a subset of parameters for the image encoder using the text embeddings; Generating image embeddings based on pixels of the image data using the image encoder, while applying the subset of parameters; Minimizing loss between image embeddings and text embeddings; and Updating the machine learning model and neural network using loss. [2] Computer-implemented method according to claim 1, wherein: the machine learning model is a vision-language model; the neural network includes a hypernetwork; and The hypernetwork comprises a non-causal transformer model that includes transformer layers which generate at least the subset of parameters. [3] Computer-implemented method according to claim 1, wherein the loss includes a contrastive loss or a sigmoid-based loss. [4] Computer-implemented method according to claim 1, wherein the subset of parameters includes normalization parameters. [5] Computer-implemented method according to claim 1, wherein the subset of parameters includes a single group of weights for the image encoder that are associated with a batch of text embeddings. [6] Computer-implemented method according to claim 1, wherein: the image encoder includes a different subset of parameters, the other subset of parameters is not updated according to the output of the neural network. [7] Computer-implemented method according to claim 1, wherein the total number of all parameters of the image encoder is less than 10 million parameters. [8] Computer-implemented method according to claim 1, wherein the total number of all parameters of the image encoder is less than the total number of all parameters of the text encoder. [9] Computer-implemented method according to claim 1, further comprising: Obtaining a set of class data for an image classification task; Generating class embeddings using the text encoder and the set of class data; Generate, using the neural network, at least an updated subset of parameters for the image encoder; and Outputting an image classifier that includes the image encoder with the updated subset of parameters, where the image classifier uses the class embeddings to perform the image classification task. [10] Computer-implemented method according to claim 9, further comprising: Inserting the image classifier into an edge device, the edge device can be controlled by means of the image classification task performed by the image classifier. [11] System, encompassing: one or more processors; one or more computer memories in data communication with the one or more processors, wherein computer-readable data is stored on the one or more computer memories, wherein the computer-readable data includes instructions which, when executed by one or more processors, cause the one or more processors to perform a method for training a machine learning model, which includes an image encoder and a text encoder, wherein the method includes: Receiving data pairs containing image data and text data, where the text data describes the corresponding image data of a respective digital image; Generating text embeddings based on the text data using the text encoder; Generating, using a neural network, at least a subset of parameters for the image encoder using the text embeddings; Generating image embeddings based on pixels of the image data using the image encoder, while applying the subset of parameters; Minimizing loss between image embeddings and text embeddings; and Updating the machine learning model and neural network using loss. [12] System according to claim 11, wherein: the machine learning model is a vision-language model; the neural network includes a hypernetwork; and The hypernetwork includes a non-causal transformer model that contains transformers which generate at least the subset of parameters. [13] System according to claim 11, wherein the loss includes a contrastive loss or a sigmoid-based loss. [14] System according to claim 11, wherein the subset of parameters includes normalization parameters. [15] System according to claim 11, wherein the subset of parameters includes a single group of weights for the image encoder that are associated with a batch of text embeddings. [16] System according to claim 11, wherein: the image encoder includes a different subset of parameters, the other subset of parameters is not updated according to the output of the neural network. [17] System according to claim 11, wherein the total number of all parameters of the image encoder is less than 10 million parameters. [18] System according to claim 11, wherein the size of the image encoder is smaller than the size of the text encoder. [19] System according to claim 11, wherein the method further comprises: Obtaining a set of class data for an image classification task; Generating class embeddings using the text encoder and the set of class data; Generate, using the neural network, an updated set of parameters for the image encoder; and Outputting an image classifier that includes the image encoder with the updated set of parameters, where the image classifier uses the class embeddings to perform the image classification task. [20] System according to claim 19, further comprising: Inserting the image classifier into an edge device, the edge device can be controlled by means of the image classification task performed by the image classifier.

Images