EMBALAGEM DE SEQUÊNCIA PARA TREINAMENTO DE REDES NEURAIS DE PROCESSAMENTO DE IMAGEM
By stacking images with varying resolutions and aspect ratios into a single input sequence during training, the method improves neural network efficiency and inference performance, addressing the limitations of conventional fixed-resolution training schemes.
Patent Information
- Application Number
- BR112025025164
- Authority / Receiving Office
- BR · BR
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-17
- Filing Date
- 2024-05-17
- Publication Date
- 2026-11-10
AI Technical Summary
Conventional neural networks require input images to be resized to a fixed resolution and aspect ratio before training, limiting their ability to process variable resolution images efficiently and effectively.
A training method that stacks multiple images with varying resolutions and aspect ratios into a single input sequence, allowing for improved training efficiency and inference performance by incorporating techniques like resolution sampling and token discarding.
The method enables neural networks to process variable resolution images with fewer computational resources, enhancing training efficiency and inference performance while maintaining accuracy.
Abstract
Description
1 / 29 SEQUENCE PACKAGING FOR TRAINING IMAGE PROCESSING NEURAL NETWORKS CROSS-REFERENCE TO RELATED REQUESTS
[0001] This application claims the benefit of priority from US Provisional Patent Application No. 63 / 467,296, filed May 17, 2023, the contents of which are incorporated herein by reference. FUNDAMENTALS
[0002] This descriptive report refers to neural network training.
[0003] Neural networks are machine learning models that employ one or more layers of nonlinear units to predict an output for a given input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input for the next layer in the network, that is, the next hidden layer or the output layer. Each layer of the network generates an output from a given input according to the current value inputs of a respective set of parameters. SUMMARY
[0004] This descriptive report describes a system, implemented as computer programs on one or more computers in one or more locations, that trains an image processing neural network.
[0005] An image processing neural network is a neural network that receives an input image and processes the input image to generate an output for the image.
[0006] Specific aspects of the subject matter described in this descriptive report can be implemented in order to achieve one or more of the following advantages.
[0007] Vision Transformers (ViTs) and other neural networks, for example, Vision Transformer variants, MLP-Mixer type neural networks and so on, which process image patch sequences have demonstrated very strong performance in several vision tasks. Petition 870250104979, dated 11 / 17 / 2025, page 16 / 57 2 / 29 computational. However, these neural networks generally require that, during training, the input images be resized to a fixed resolution and a fixed aspect ratio, for example, a square aspect ratio, and then divided into a fixed number of patches before they are processed by the neural network.
[0008] This descriptive report, on the other hand, describes an alternative approach where, during training, multiple patches of multiple different images are “stacked” into a single input sequence. That is, a single input sequence during training may include patches of multiple different input images, and these different input images may have different resolutions, different aspect ratios, or both.
[0009] Training the neural network in this way can make it possible to process variable resolution images into inference, can improve training efficiency and downstream inference performance, and allows the incorporation of several other techniques that can further improve training efficiency. Examples of these techniques include resolution sampling and discarding randomly sampled tokens.
[0010] In other words, by modifying the training scheme of an image processing neural network, as described in this descriptive report, the image processing neural network can be trained to have improved inference time performance while remaining capable of accurately processing images of different resolutions and aspect ratios. Furthermore, the neural network can achieve this improved performance while consuming fewer computational resources during training.
[0011] The details of one or more embodiments of the subject matter of this descriptive report are presented in the attached drawings and in the description below. Other characteristics, aspects and advantages of the subject matter will become apparent from the description, figures and claims. BRIEF DESCRIPTION OF THE DRAWINGS
[0012] Figure 1 shows an example neural network system. Petition 870250104979, dated 11 / 17 / 2025, page 17 / 57 3 / 29
[0013] Figure 2 is a flowchart of an example process for performing a training step during the training of the image processing neural network.
[0014] Figure 3 is a flowchart of an example process for generating an input sequence when token discarding is employed.
[0015] Figure 4 is a flowchart of an example process for obtaining a batch when resolution sampling is employed.
[0016] Figure 5 shows an example of generating an input sequence and processing the input sequence using an image processing neural network.
[0017] Figure 6 shows an example of the performance of the techniques described.
[0018] Similar reference numbers and designations in the various drawings indicate similar elements. DETAILED DESCRIPTION
[0019] Figure 1 shows an example neural network system 100. Neural network system 100 is an example of a system implemented as computer programs on one or more computers in one or more locations, where the systems, components, and techniques described below can be implemented.
[0020] In particular, system 100 is a system that trains an image processing neural network 110 on training data 120.
[0021] The image processing neural network 110 is a neural network that receives an input image and processes the input image to generate an output for the image.
[0022] For example, the output for the image could be an image embedding of the input image in an embedding space. An “embedding,” as used in this descriptive report, is a vector of numeric values, for example, floating-point values or other values, with a predetermined dimensionality. The space of possible vectors with the Petition 870250104979, dated 11 / 17 / 2025, page 18 / 57 4 / 29 predetermined dimensionality is called the “embedding space”. In other words, an embedding of an image is an encoding of the image.
[0023] As another example, the output for the image could be a classification output for a classification task for the input image. That is, the output includes a respective score for each of a set of object categories, where the score for an object category represents the probability that the image depicts an object that belongs to that object category.
[0024] After being trained, the image processing neural network 110 can be adapted to one or more downstream tasks. In other words, the training performed by the system 100 on the training data 120 can be called a “pre-training” stage that is performed in order to improve how well and how quickly the neural network 110 can be adapted to one or more downstream tasks after being trained on the training data 120.
[0025] To adapt the image processing neural network 110, the system 100 can train a downstream neural network 130 that includes at least some of the layers of the image processing neural network 110 on training data for the downstream task.
[0026] For example, the downstream task could be an image classification task, as described above.
[0027] As another example, the downstream task could be object detection, for example, open-vocabulary object detection, where the output for a given image identifies the locations of one or more bounding boxes in the image and, for each bounding box, a category to which an object depicted in each bounding box belongs. An example of an object detection task is an open-vocabulary object detection task, where the set of possible categories may be different for different inputs and is specified by the embedding of category labels. Petition 870250104979, dated 11 / 17 / 2025, page 19 / 57 5 / 29 generated by a text encoder neural network. For example, in open vocabulary object detection, for each bounding box, the system can select the category label embedding that has the highest similarity, for example, in terms of cosine or dot product similarity, with a bounding box embedding generated by the downstream neural network.
[0028] As another example, the downstream task could be image segmentation. That is, the neural network could be configured to generate an element-level classification output (e.g., a pixel-level classification output) that includes, for each element in the input network, a corresponding score that corresponds to each of the multiple categories. For a given element (e.g., for a given pixel), the score for a category indicates a probability that the element belongs to the category. In some cases, the categories may be classes of objects, and an element may belong to a category if it is part of an object included in the object class corresponding to the category.
[0029] As another example, the downstream task could be image depth prediction. In a depth prediction task, the output generated by the neural network identifies, for each pixel in the image, a predicted depth of the scene at that pixel.
[0030] As another example, the downstream task could be a video comprehension task. That is, the neural network could be configured to process a sequence of video frames to generate an output that characterizes the video frames, for example, characterizing whether the video frames depict a person or other agent performing a specific action, generating a caption that describes the semantic content of the video, classifying one or more objects in the video, and so on.
[0031] As another example, the downstream task could be a multimodal text and vision task, for example, image captioning, where the input is an image and the output is a text caption that describes the image. Petition 870250104979, dated 11 / 17 / 2025, page 20 / 57 6 / 29 input, or visual question response, where the input is an image or video and a question about the image or video, and the output is an answer to the question.
[0032] The image processing neural network 110 can have any appropriate architecture that processes a sequence of tokens representing an image to generate the image output.
[0033] For example, neural network 110 may have an architecture that includes multiple self-attention network blocks that each perform self-attention to update tokens in the sequence. Examples of such architectures include Vision Transformers (ViTs) and other ViT variants. As a specific example, the neural network may have the architecture described by Dosovitskiy, et al., An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, arXiv:2010.11929.
[0034] As another example, neural network 110 may have an architecture that includes multiple network blocks that perform a different type of operation to update tokens in the sequence. An example of such an architecture is an MLP-mixer type architecture. As a specific example, the neural network may have the architecture described by Tolstikhin, et al., MLP-Mixer: An all-MLP Architecture for Vision, arXiv:2105.01601.
[0035] In any of the above examples, the 110 neural network may include a sequence of network blocks that each update the tokens in the input sequence. The 110 neural network may also include one or more additional neural network layers that process the outputs of the last network block in the sequence to generate the training output.
[0036] When adapting neural network 110 for a new downstream task, system 100 or another adaptation system can replace one or more additional neural network layers with a set of additional neural network layers that are specific to the downstream task. System 100 can then train the resulting downstream neural network on training data for the downstream task, either by keeping the network blocks fixed and training Petition 870250104979, dated 11 / 17 / 2025, page 21 / 57 7 / 29 only the new additional layers or training both the new additional layers and the network blocks.
[0037] More specifically, given an input image, neural network 110 divides the input image into patches. Neural network 110 then generates a respective embedding for each patch and processes a sequence that includes the embeddings for the patches to generate the training output for the input image.
[0038] Splitting an image into patches and generating patch embeds will be described in more detail below.
[0039] In general, the training data 120 includes multiple different training examples, where each training example includes at least one training image and, optionally, other data, for example, a classification label or a corresponding text string.
[0040] The training images in the training examples generally have varying resolutions and aspect ratios. However, conventional training schemes for training these neural networks generally require that the training images be resized to a fixed resolution and a fixed aspect ratio, for example, a square aspect ratio, and then divided into a fixed number of patches before being processed by the neural network 110.
[0041] In order to improve upon these conventional training schemes and as described in more detail below, system 100 trains neural network 110 by “stacking” multiple images into any given “stacked” input sequence 140 which is processed by neural network 110 during training.
[0042] Training neural network 110 in this way has many implications for training neural network 110 and subsequently for fine-tuning and using (at least a portion of) neural network 110 to perform inference. Petition 870250104979, dated 11 / 17 / 2025, page 22 / 57 8 / 29
[0043] For example, on a fixed computational budget, the described training scheme consistently outperforms conventional approaches. For example, the described training scheme can result in a neural network 110 that matches the performance of a Vision Transformer (ViT) trained using prior state-of-the-art techniques with 4x less computation. As one example, since sequence packing results in multiple images being combined within the same input sequence, there is a substantial increase in the number of training examples processed within the allocated computational budget, which contributes to the performance increase. As another specific example, sequence packing coupled with variable resolution inputs and variable token discarding (described below) enables the system 100 to process five times more images during training than in conventional schemes.Additionally, this improved efficiency extends to the fine-tuning process, where similar schemes can be applied.
[0044] Furthermore, by exposing neural network 110 to multiple resolutions during both pretraining and fine-tuning, a single model demonstrates excellent performance when evaluated at various resolutions, significantly benefiting neural network 110 in terms of inference costs compared to neural networks trained using conventional schemes.
[0045] Figure 2 is a flowchart of an example process 200 to perform a training step during the training of the image processing neural network. For convenience, process 200 will be described as being performed by a system of one or more computers located in one or more locations. For example, a neural network system, for example, neural network system 100 of Figure 1, appropriately programmed, can perform process 200.
[0046] In general, the system can perform image processing neural network training through multiple steps of Petition 870250104979, dated 11 / 17 / 2025, page 23 / 57 9 / 29 training. In each training step, the system obtains a batch of training examples and trains the image processing neural network on the training examples in the batch, that is, performing one iteration of the process 200.
[0047] To perform the training, the system obtains a batch of training examples, each training example including a corresponding training image (step 202).
[0048] In some cases, the training example includes only the image. In other cases, however, the training example includes other information, for example, a corresponding text string that describes the image or data that identifies a label for the image.
[0049] In general, the training sample set may include images with varying resolutions and aspect ratios.
[0050] In some implementations, the system trains the neural network at the original resolutions of the training images in the batch.
[0051] In some other implementations, the system uses resolution sampling when generating the batch of training images.
[0052] More specifically, in conventional training schemes, there is a tension between higher throughput during training (training on smaller images) and higher performance (training on larger images, to enable the use of high-resolution images at the time of evaluation). As a result, models are sometimes pre-trained at a lower resolution and fine-tuned at a higher resolution. However, when using resolution sampling, the techniques described are much more flexible, since they can allow resolution mixing training by sampling from a distribution of image sizes while maintaining the original aspect ratio of each image. This allows both higher throughput and exposure to large images, yielding substantially improved performance, for example, over equivalent ViTs (in terms of model size and training duration). Petition 870250104979, dated 11 / 17 / 2025, page 24 / 57 10 / 29
[0053] Resolution sampling is described in more detail below with reference to Figure 4.
[0054] For each training image in the training examples, the system divides the training image into a respective plurality of patches and generates a respective token that represents each of the patches (step 204).
[0055] To divide a training image into patches, the system can divide each image into regions of fixed size. Although the regions have a fixed size, since different images have different resolutions, the different images can be divided into different numbers of patches.
[0056] In general, to generate the tokens that represent the patches, the system generates a patch embedding of the patch and generates a positional embedding of a patch position within the corresponding training image.
[0057] The system then combines the patch embedding and the positional embedding to generate the respective token that represents the patch. For example, the system can sum, average, or concatenate the patch embedding and the positional embedding.
[0058] To generate the patch embedding of any given patch, the system can process the intensity values of the pixels in the patch using a patch embedding subnetwork to generate the patch embedding. For example, the patch embedding subnetwork can be a single linear projection layer, it can be a multi-layer subnetwork, for example, an MLP, or have a different appropriate architecture.
[0059] In general, the patch embedding subnetwork is trained together with the rest of the image processing neural network, for example, it is trained using gradients of the loss function, computed as described below.
[0060] The system can generate the positional embedding of the position of a Petition 870250104979, dated 11 / 17 / 2025, page 25 / 57 11 / 29 data patch within the corresponding training image in any one of several ways.
[0061] For example, the system can use one-dimensional (1-D) positional embeddings that map each coordinate to a position in a flattened 1-D representation of the image and are learned along with the rest of the neural network.
[0062] As another example, the system can use two-dimensional (2-D) positional embeddings that are learned along with the rest of the neural network.
[0063] That is, in this example, a corresponding embedding for each index in a grid [maxLen, maxLen] is learned, where the grid is indexed with the coordinates (x, γ) of each patch and maxLen is a predefined maximum length of any given dimension of an image. This allows for variable aspect ratios, with resolutions up to R = P · maxLen, where P is the number of patches in any given image. However, every combination of coordinates (x, γ) must be seen during training. For example, the coordinate of a given patch for a given dimension can represent the patch index along that dimension in terms of patch numbers, for example, so that x = 2 will be the second patch along the x dimension.
[0064] However, although 2D embeddings support variable aspect ratios, none of the above encoding schemes readily extrapolate to unseen resolutions. To address this issue, in some implementations, the system uses “factorized” positional embeddings.
[0065] In this scheme, the system maintains a separate set of embeddings for each of a set of multiple coordinates xe and a separate set of embeddings for each of a set of multiple coordinates γ.
[0066] In some cases, the embeddings are absolute embeddings, so that the embedding of a given coordinate is a function of Petition 870250104979, dated 11 / 17 / 2025, page 26 / 57 12 / 29 absolute index of the coordinate within the image.
[0067] In some other cases, the embeddings are fractional embeddings, so that the embedding of a given coordinate is a function of the ratio between the absolute index of the coordinate within the image and the side length of the image along the corresponding dimension, that is, the number of patches along the corresponding dimension for the given image.
[0068] When using factored positional embeddings, to determine the positional embedding for a given patch, the system generates a first embedding of an x-coordinate of the patch's position within the corresponding training image, for example, by mapping the absolute index of the coordinate to the first embedding, or by mapping the ratio between the coordinate within the image and the image's side length along the x-dimension to the first embedding.
[0069] The system generates a second embedding of a γ coordinate of the patch position within the corresponding training image, for example, by mapping the absolute index of the coordinate to the second embedding, or by mapping the ratio between the coordinate within the image and the side length of the image along the γ dimension to the second embedding.
[0070] The system then combines the first and second embeddings to generate the positional embedding of the patch position.
[0071] For example, the system can add the first and second embeds to generate the positional embed of the patch position.
[0072] As another example, the system can stack, that is, concatenate, the first and second embeds to generate the positional embed of the patch position.
[0073] As another example, the system can multiply the first and second embeddings into elements to generate the positional embedding of the patch position.
[0074] When using the factored scheme, the set of incorporations for the Petition 870250104979, dated 11 / 17 / 2025, page 27 / 57 13 / 29 dimension xeo The set of embeddings for dimension γ can be learned along with the rest of the neural network, for example, as learned pattern embeddings or learned positional Fourier embeddings, or they can be determined before training, for example, as sinusoidal embeddings.
[0075] The system generates one or more input sequences (step 206). In general, the one or more input sequences collectively represent all the images in the batch.
[0076] Each input sequence corresponds to two or more of the training images in the batch and is a token sequence that includes tokens representing patches of the two or more corresponding training images.
[0077] In other words, the system generates the input sequences using sequence packing, “packaging” tokens from multiple images into the same input sequence.
[0078] In general, since the images in the batch can have varying resolutions and aspect ratios, a sequence can include tokens from two different images with two different resolutions, two different aspect ratios, or both.
[0079] In particular, as described above, the number of tokens representing an image may depend on the image resolution, so that images with different resolutions are represented by different numbers of tokens. Thus, any given input sequence may include different numbers of tokens for two different images with two different resolutions.
[0080] In some implementations, the system employs a token discarding scheme when generating each input sequence. As part of employing token discarding, the system may determine, for each image, the possibility of removing any tokens representing the image from the input sequence and then generating the input sequence with only those tokens. Petition 870250104979, dated 11 / 17 / 2025, page 28 / 57 14 / 29 tokens that were not removed from each of the two or more input images.
[0081] This will be described in more detail below with reference to Figure 3.
[0082] The system can determine which images in a given batch are represented in which input sequence in any one of several ways.
[0083] For example, in some implementations, each input sequence must include the same maximum number of tokens, that is, in order to optimize the performance of the hardware on which the neural network is being trained, for example, tensor processing units (TPUs) or other ASICs that are optimized for fixed-size inputs.
[0084] In these implementations, the system can generate input sequences using a greedy approach in which the system adds images to the first sequence with sufficient remaining space. That is, the system can traverse the images in the batch, for example, according to a random ordering of the images, and can determine if the tokens representing the image can “fit” within any already generated partially complete input sequence. If so, the system adds the image to one of the partially complete input sequences. If not, the system places the image in a new sequence. Since no image can fit into any of the generated sequences, the sequences are padded with padding tokens (“masking”), yielding fixed sequence lengths.
[0085] As another example, instead of using masking tokens, when generating a given input sequence, the system can dynamically choose the resolution or token drop rate of the final image in the sequence to exactly fit the remaining tokens in the given input sequence.
[0086] The system then processes each input sequence using the image processing neural network to generate a corresponding output. Petition 870250104979, dated 11 / 17 / 2025, page 29 / 57 15 / 29 of training for each corresponding training image (step 208).
[0087] For example, when the image processing neural network includes a sequence of self-attention layers, the system processes the input sequence through the sequence of self-attention layers to update the tokens in the input sequence and then generates the training output for each training image from the updated tokens.
[0088] An example of this is described below with reference to Figure 5.
[0089] The system trains the image processing neural network on a loss function for the task using the training outputs for the training images in the batch (step 210). For example, the loss function can be a contrastive loss function or a classification loss function, for example, a loss function with a cross-entropy term.
[0090] More specifically, in some cases, the system trains the neural network through contrastive learning. In these cases, the training output generated by the image processing neural network for each image is an image embedding of the image into an embedding space.
[0091] Additionally, the image processing neural network is trained together with a text encoder neural network that has text encoder neural network parameters and is configured to process a text segment to generate a text embedding of the text segment in the embedding space. For example, the text encoder neural network could be a transformer neural network, a recurrent neural network (RNN), or another appropriate type of text processing neural network.
[0092] Thus, in these cases, each training example also includes a corresponding training text segment, for example, a text segment that has been determined to be semantically similar to the training image in the training example. For example, within a given training example, the text segment might be a text note from Petition 870250104979, dated 11 / 17 / 2025, page 30 / 57 16 / 29 An image from a set of manually or automatically generated image notes, or it may be alternative text associated with the image in an alternative text dataset. Alternative text is text that is displayed in place of an image on a web page, for example, if the image cannot be rendered properly or otherwise fails to load. For example, the system may obtain the alternative text data from data maintained by an internet search engine or other software that automatically crawls web pages on the internet.
[0093] To train the text encoder and image processing neural networks in these cases, the system can, for each training text segment, process the training text segment using the text encoder neural network to generate a training text embedding of the training text segment.
[0094] The system can then train the text encoder and image processing neural networks on a contrastive loss function that is based on similarities between the training text embeddings and the training image embeddings.
[0095] In some other cases, the system trains the image processing neural network on an image classification task. In these cases, the training output generated by the image processing neural network may be a classification output, and each training example includes a fundamental truth classification for the training image in the training example. Thus, the system trains the image processing neural network on a loss function that includes, for each training example, a classification loss relative to the fundamental truth classification for the training image, for example, a cross-entropy or other classification loss.
[0096] As described above, after repeatedly performing 200 iterations of the process to train the image processing neural network, the system can then adapt the image processing neural network to perform Petition 870250104979, dated 11 / 17 / 2025, page 31 / 57 17 / 29 a downstream computer vision task. Examples of such tasks include image classification, object detection, image segmentation, depth prediction, video understanding, and multimodal vision and text tasks.
[0097] Figure 3 is a flowchart of an example process 300 for generating an input sequence when the system performs token discarding. For convenience, process 300 will be described as being performed by a system of one or more computers located in one or more locations. For example, a neural network system, such as neural network system 100 in Figure 1, appropriately programmed, can perform process 300.
[0098] Token discarding (i.e., the random omission of input patches during training) was developed to accelerate training. However, typically, the same proportion of tokens is discarded from all examples. By using sequence packing, the system enables continuous token discarding, where the token discard rate can be varied per image. This enables the faster throughput benefits of token discarding while still allowing the model to process some complete images, reducing the training / inference discrepancy and improving performance at inference time. Additionally, with packing, the discard distribution can vary throughout training, further improving training performance.
[0099] The system selects two or more of the training images from the batch of training examples (step 302). For example, the system might select the training images that will be represented in the training sequence in a greedy manner, as described above.
[0100] The system determines, for each of the two or more training images, a respective token discard rate (step 304).
[0101] In some cases, the respective token discard rate is the same for all training images in the batch. In these cases, the system may Petition 870250104979, dated 11 / 17 / 2025, page 32 / 57 18 / 29 determine the token discard rate for the batch of training images in any one of several ways.
[0102] As an example, the system can randomly sample the respective token discard rate for the batch from a set of possible token discard rates. For example, the system can sample the token discard rate from a Beta distribution or other distribution over a range of possible token discard rates.
[0103] As another example, the system can determine the respective token discard rate for the batch according to a training stage index between training stages performed during training. That is, the system can maintain a function that maps the training stage index to a token discard rate.
[0104] In some other cases, different images in a batch may have different token discard rates.
[0105] For example, the system can determine the respective token discard rate for a given training image in the batch according to an index of the training image among training images processed during training. That is, the system can maintain a function that maps the training image index to a token discard rate.
[0106] As another example, the system can determine the token drop rate based on the resolution of the training image. For example, the system can sample the token drop rate for a given image from a distribution that depends on the resolution of the training image, for example, which has a mean that is based on the resolution of the training image.
[0107] For each of the two or more training images, the system determines, according to the token discard rate for the training image, the possibility of selecting any of the tokens representing patches from the two or more training images for removal. Petition 870250104979, dated 11 / 17 / 2025, page 33 / 57 19 / 29 (stage 306).
[0108] The system generates an input sequence that includes only the tokens for the two or more images that were not selected for removal (step 308).
[0109] Figure 4 is a flowchart of an example 400 process for obtaining a batch of training examples when the system employs resolution sampling. For convenience, the 400 process will be described as being performed by a system of one or more computers located in one or more locations. For example, a neural network system, such as the neural network system 100 in Figure 1, appropriately programmed, can perform the 400 process.
[0110] The system samples an original batch of original training examples (step 402). As described above, each original training example includes an original training image, where each original training image has a respective original resolution and a respective original aspect ratio. For example, the system may sample the original batch from a larger set of training examples that is available in the system for use in training the neural network.
[0111] The system modifies the original images in the original training examples (step 404).
[0112] More specifically, for each original training image, the system samples a resolution for the original training image and then modifies the original training image to have the sampled resolution while maintaining the original aspect ratio of the original training image. That is, the total number of pixels in the original image is re-sampled to match the sampled resolution while the aspect ratio is preserved.
[0113] As an example, the system can sample the resolution from a fixed set of resolutions, for example, by sampling the resolution from a uniform distribution over the resolutions in the fixed set. Petition 870250104979, dated 11 / 17 / 2025, page 34 / 57 20 / 29
[0114] The system generates a batch of training examples that includes the modified images (step 406). That is, the training example corresponding to any given original training example includes the modified training image generated from the original training image in the training example and any other information that was in the original training example, for example, a training text if the neural network is being trained through contrastive learning or a fundamental truth label if the neural network is being trained to perform classification.
[0115] Figure 5 shows an example 500 of generating and processing an input sequence during a given training step during the training step.
[0116] In particular, in example 500, the image processing neural network is a neural network that includes one or more layers of self-attention that each apply self-attention.
[0117] As shown in example 500, the system selects three input images 502 (image 1, image 2 and image 3).
[0118] The system then “patches” each of the input images 502 to split each of the images 502 into patches.
[0119] After splitting the images into patches, the system can perform token 506 discarding on each input image 502 to determine if any of the tokens representing patches from any of the images should be removed.
[0120] The system then generates an input sequence 510 (“stacked sequence”) that includes only the tokens that were not removed from the three input images. In particular, the input sequence 510 includes a respective subsequence for each of the three input images. More specifically, the input sequence 510 includes a concatenation of a plurality of subsequences that each include tokens representing patches of a respective image among the three training images. Petition 870250104979, dated 11 / 17 / 2025, p. 35 / 57 21 / 29
[0121] In example 500, since the concatenation of the plurality of subsequences includes less than the maximum number of tokens, the system appended two masking tokens to the “padding” portion of input sequence 510, so that input sequence 510 has the maximum number of tokens.
[0122] The system then processes the input sequence 510 using a sequence of self-attention blocks 520. The self-attention network blocks 520 each apply self-attention to the tokens in the input sequence 510 to update at least a subset of the tokens in the input sequence. For example, the masking tokens in the “filling” section of the input sequence 510 can be masked so that they are not updated by the self-attention layer.
[0123] To ensure that tokens for a given image do not influence the final output for other images, for each of the 520 self-attention network blocks, self-attention is masked so that tokens for each corresponding image are updated only by using tokens for the corresponding image and not tokens for any other corresponding images or padding.
[0124] In example 500, once the neural network has processed the sequence through the blocks 520, the neural network applies a clustering operation 530, for example, global mean clustering or another type of clustering, to the tokens to generate a respective clustering representation for each of the three images that correspond to the input sequence. Once the clustering representations have been generated, the neural network can generate the respective training output for each image using the clustering representation for the image.
[0125] As an example, for contrastive learning tasks, the system can use the clustering representation as the embedding of the input image or it can apply one or more transformations, for example, learned projections or other learned operations, to the representation of Petition 870250104979, dated 11 / 17 / 2025, page 36 / 57 22 / 29 grouping to generate the incorporation.
[0126] As another example, for classification tasks, the system can process the clustering representation using an output subnetwork to generate the classification output. For example, the output subnetwork might include one or more fully connected layers followed by a softmax layer.
[0127] To ensure that the tokens for a given image do not influence the final output for other images, the 530 grouping operation is masked, so that the grouping representation for each corresponding image is generated only using the tokens for the corresponding image and not the tokens for any other corresponding images, that is, only the tokens for the corresponding image and not the tokens for any other corresponding images or padding are grouped to generate the grouping representation.
[0128] Figure 6 shows example 600 of the performance of the techniques described. In particular, example 600 shows the performance of several architectures trained using the techniques described in relation to the same architectures trained using a ViT training technique.
[0129] As can be seen from Figure 6, the techniques described outperform ViT models across several dimensions: model accuracy given pretraining hours, fine-tuned accuracy at a given pretraining cost, and fine-tuned accuracy at a given inference cost.
[0130] This descriptive report uses the term “configured” in connection with computer systems and program components. A system of one or more computers being configured to perform particular operations or actions means that the system has installed on it software, firmware, hardware, or a combination thereof that, in operation, causes the system to perform the operations or actions. One or more computer programs being configured to perform particular operations or actions means that the one or more programs include instructions that, when executed by Petition 870250104979, dated 11 / 17 / 2025, page 37 / 57 23 / 29 data processing device, they cause the device to perform operations or actions.
[0131] Embodiments of the matter and functional operations described in this descriptive report may be implemented in digital electronic circuits, in tangibly embedded computer software or firmware, in computer hardware, including the structures disclosed in this descriptive report and their structural equivalents, or in combinations of one or more of them. Embodiments of the matter described in this descriptive report may be implemented as one or more computer programs, for example, one or more computer program instruction modules encoded in a tangible non-transient storage medium for execution by, or to control the operation of, data processing apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random-access or serial memory device, or a combination of one or more of them.Alternatively or additionally, the program instructions may be encoded in an artificially generated propagated signal, for example, a machine-generated electrical, optical, or electromagnetic signal that is generated to encode information for transmission to a suitable receiving device for execution by a data processing device.
[0132] The term data processing device refers to data processing hardware and encompasses all types of devices, apparatus, and machines for data processing, including, for example, a programmable processor, a computer, or multiple processors or computers. The device may also be, or include, special-purpose logic circuits, for example, an FPGA (field-programmable gate array) or an ASIC (application-specific integrated circuit). The device may optionally include, in addition to hardware, code that creates an execution environment for computer programs, for example, code that constitutes processor firmware, a protocol stack, a Petition 870250104979, dated 11 / 17 / 2025, page 38 / 57 24 / 29 database management system, an operating system, or a combination of one or more of them.
[0133] A computer program, which may also be referred to or described as a program, software, a software application, an app, a module, a software module, a script, or code, may be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and may be implemented in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but does not have to, correspond to a file in a file system. A program may be stored in a portion of a file that contains other programs or data, for example, one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, for example, files that store one or more modules, subprograms, or portions of code.A computer program can be implemented to run on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a data communication network.
[0134] In this descriptive report, the term database is used broadly to refer to any collection of data: the data need not be structured in any particular way, nor structured in any way, and may be stored on storage devices in one or more locations. Thus, for example, an index database may include multiple collections of data, each of which may be organized and accessed differently.
[0135] Similarly, in this descriptive report, the term tool is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more functions. Petition 870250104979, dated 11 / 17 / 2025, pp. 39 / 57 25 / 29 specific. Generally, a tool will be implemented as one or more software modules or components, installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a specific tool; in other cases, multiple tools may be installed and run on the same computer or computers.
[0136] The processes and logical flows described in this descriptive report can be performed by one or more programmable computers running one or more computer programs to perform functions operating on input data and generating output. The processes and logical flows can also be performed by special-purpose logic circuits, for example, an FPGA or an ASIC, or by a combination of special-purpose logic circuits and one or more programmed computers.
[0137] Computers suitable for running a computer program may be based on general-purpose or special-purpose microprocessors or both, or any other type of central processing unit. Generally, a central processing unit will receive instructions and data from a read-only memory or a random-access memory or both. The essential elements of a computer are a central processing unit to perform or execute instructions and one or more memory devices to store instructions and data. The central processing unit and memory may be supplemented by, or incorporated into, special-purpose logic circuits. In general, a computer will also include, or be operationally coupled to receive data from, or transfer data to, or both, one or more mass storage devices to store data, for example, magnetic disks, magnetic optical disks, or optical disks.However, a computer does not need to have such devices. Furthermore, a computer can be incorporated into another device, for example, a cell phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or... Petition 870250104979, dated 11 / 17 / 2025, pp. 40 / 57 26 / 29 a portable storage device, for example, a Universal Serial Bus (USB) flash drive, to name just a few.
[0138] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, memory media and devices, including, by way of example, semiconductor memory devices, for example, EPROM, EEPROM and flash memory devices; magnetic disks, for example, internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.
[0139] To provide interaction with a user, the modalities of the subject matter described in this descriptive report may be implemented on a computer with a display device, for example, a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, to display information to the user, and a keyboard and a pointing device, for example, a mouse or a trackball, by which the user may provide input to the computer. Other types of devices may be used to provide interaction with a user as well; for example, feedback provided to the user may be any form of sensory feedback, for example, visual feedback, auditory feedback, or tactile feedback; and user input may be received in any form, including acoustic, speech, or tactile input.Furthermore, a computer can interact with a user by sending documents to and receiving documents from a device used by the user; for example, by sending web pages to a web browser on a user's device in response to requests received from the web browser. Also, a computer can interact with a user by sending text messages or other forms of messaging to a personal device, for example, a smartphone running a messaging application, and receiving responsive messages from the user in return.
[0140] Data processing devices for implementing machine learning models may also include, for example, units Petition 870250104979, dated 11 / 17 / 2025, pp. 41 / 57 27 / 29 special-purpose hardware accelerators for processing common and computationally intensive training or production machine learning workloads, i.e., inference workloads.
[0141] Machine learning models can be implemented and deployed using a machine learning framework, for example, a TensorFlow framework or a Jax framework.
[0142] The material described in this descriptive report may be implemented in a computing system that includes a back-end component, for example, a data server, or that includes a middleware component, for example, an application server, or that includes a front-end component, for example, a client computer that has a graphical user interface, a web browser, or an application through which a user can interact with an implementation of the material described in this descriptive report, or any combination of one or more of these back-end, middleware, or front-end components. The system components may be interconnected by any form or means of digital data communication, for example, a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), for example, the Internet.
[0143] A computer system may include clients and servers. A client and a server are generally remote from each other and typically interact through a communication network. The client-server relationship emerges by virtue of computer programs running on their respective computers and a client / server relationship with each other. In some embodiments, a server transmits data, for example, an HTML page, to a user device, for example, for purposes of displaying data and receiving user input from a user interacting with the device, which acts as a client. The data generated on the user's device, for example, a result of the user's interaction, may be received on the server from the device. Petition 870250104979, dated 11 / 17 / 2025, pp. 42 / 57 28 / 29
[0144] Although this descriptive report contains many specific implementation details, these should not be interpreted as limitations on the scope of any invention or on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this descriptive report in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, several features that are described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable subcombination.Furthermore, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features of a claimed combination may, in some cases, be removed from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.
[0145] Similarly, although operations are depicted in the figures and cited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together into a single software product or packaged into multiple software products.
[0146] The particular modalities of the matter have been described. Other modalities are within the scope of the following claims. For example, the actions recited in the claims can be performed in an order Petition 870250104979, dated 11 / 17 / 2025, pp. 43 / 57 29 / 29 different and still achieve desirable results. As an example, the processes represented in the attached figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing can be advantageous. Petition 870250104979, dated 11 / 17 / 2025, pp. 44 / 57
Claims
1 / 6 CLAIMS 1. A method performed by one or more computers for training an image processing neural network that has image processing neural network parameters to perform a first task, wherein the method is characterized in that it comprises, in each of a plurality of training steps: obtaining a batch of training examples, each training example comprising a respective training image; for each training image: dividing the training image into a respective plurality of patches; and generating a respective token representing each of the patches; generating one or more input sequences, wherein each input sequence corresponds to two or more of the training images in the batch and is a sequence of tokens comprising tokens representing patches of two or more corresponding training images;Process each input sequence using the image processing neural network to generate a corresponding training output for each training image; and train the image processing neural network on a first loss function for the first task using the training outputs for the training images in the batch.
2. Method, according to claim 1, characterized in that: the training output generated by the image processing neural network for each image is an image embedding of the image into an embedding space; the image processing neural network is trained together with a text encoder neural network that has text encoder neural network parameters and is configured to process a text segment for Petition 870250104979, dated 11 / 17 / 2025, p.45 / 57 2 / 6 generate a text embedding of the text segment in the embedding space, each training example further comprises a respective training text segment, and the method further comprises: for each training text segment: processing the training text segment using the text encoder neural network to generate a training text embedding of the training text segment; and wherein the first loss function is a contrastive loss function that is based on similarities between the training text embeddings and the training image embeddings.
3. A method according to claim 1, characterized in that the training output generated by the image processing neural network for each image is an image classification output for the image, and wherein the loss function is a classification loss relative to a fundamental truth classification for the image.
4. A method, according to any of the preceding claims, characterized in that it further comprises: after training the image processing neural network on the first loss function for the first task; adapting the image processing neural network to perform a downstream computer vision task.
5. Method according to claim 4, characterized in that the downstream computer vision task is one of: image classification, object detection, image segmentation, depth prediction, video understanding, or a text and multimodal vision task. Petition 870250104979, dated 11 / 17 / 2025, pp. 46 / 57 3 / 6 6. Method, according to claim 5, characterized in that the multimodal text and vision task is either image captioning or a visual question answering task.
7. Method, according to claim 5, characterized in that the computer vision task is open vocabulary object detection.
8. A method, according to any of the preceding claims, characterized in that the generation of one or more input sequences, wherein each input sequence corresponds to two or more of the training images in the batch and is a sequence of tokens comprising tokens representing patches of two or more corresponding training images, comprises: selecting two or more of the training images; determining, for each of the two or more training images, a respective token discard rate; and for each of the two or more training images, determining, according to the token discard rate for the training image, the possibility of selecting any of the tokens representing patches of the two or more training images for removal; and generating an input sequence comprising only the tokens for the two or more images that were not selected for removal.
9. Method according to claim 8, characterized in that the respective token discard rate is the same for all training images in the batch.
10. A method according to claim 9, characterized in that the determination, for each of two or more training images, of a respective token discard rate comprises randomly sampling the respective token discard rate for the batch from a set of possible token discard rates.
11. Method, according to claim 9, characterized by the fact that the determination, for each of two or more training images, of a respective token discard rate comprises determining the respective token discard rate for the batch according to a training step index among the training steps performed during training.
12. A method, according to any of the preceding claims, characterized in that the generation of a respective token representing each of the patches comprises: generating a patch embedding of the patch; generating a positional embedding of a patch position within the corresponding training image; and combining the patch embedding of the patch and the positional embedding of the patch to generate the respective token representing the patch.
13. Method, according to claim 12, characterized in that the patch embedding generation comprises: processing the intensity values of the pixels in the patch using a patch embedding subnetwork to generate the respective patch embedding.
14. Method, according to claim 13, characterized in that the training of the image processing neural network comprises: training the patch embedding subnetwork using gradients of the first loss function.
15. A method, according to any one of claims 12-14, characterized in that the generation of a positional embedding of a patch position within the corresponding training image comprises: generating a first embedding of an x-coordinate of the patch position within the corresponding training image; generating a second embedding of a y-coordinate of the patch position within the corresponding training image; and combining the first and second embeddings to generate the positional embedding of the patch position.
16. A method, according to any of the preceding claims, characterized in that each input sequence comprises a concatenation of a plurality of subsequences, each comprising tokens representing patches of a respective image from among two or more corresponding training images.
17. A method according to claim 16, characterized in that the generation of each input sequence comprises: determining whether the concatenation of the plurality of subsequences includes less than a maximum number of tokens; and in response to the determination that the concatenation of the plurality of subsequences includes less than a maximum number of tokens, attaching one or more masking tokens to the input sequence so that the input sequence has the maximum number of tokens.
18. A method, according to any of the preceding claims, characterized in that the image processing neural network has a plurality of self-attention network blocks, each applying self-attention to the tokens in each input sequence to update at least a subset of the tokens in the input sequence, and wherein, for each of the self-attention network blocks, the self-attention is masked so that, for each input sequence, the tokens for each corresponding image are updated only by using the tokens for the corresponding image and not the tokens for any other corresponding images.
19. A method, according to any of the preceding claims, characterized in that, for at least one of the input sequences, the two or more corresponding images include at least two images with different resolutions.
20. Method, according to any of the preceding claims Petition 870250104979, dated 11 / 17 / 2025, page 49 / 57 6 / 6, characterized in that, for at least one of the input sequences, the two or more corresponding images include at least two images with different aspect ratios.
21. A method, according to any of the preceding claims, characterized in that obtaining a batch of training examples comprises: sampling an original batch of original training examples, each original training example comprising an original training image having a respective original resolution and a respective original aspect ratio; for each original training image: sampling a resolution for the original training image; and modifying the original training image so as to have the sampled resolution while maintaining the original aspect ratio of the original training image.
22. Method according to claim 21, characterized in that two or more of the original training images have different respective original aspect ratios.
23. A system characterized in that it comprises one or more computers and one or more storage devices that store instructions which, when executed by the one or more computers, cause the one or more computers to perform the operations of the respective method, as defined in any one of claims 1-22.
24. One or more computer storage media characterized in that they store instructions which, when executed by one or more computers, cause the one or more computers to perform the operations of the respective method, as defined in any one of claims 1-22. Petition 870250104979, dated 11 / 17 / 2025, pp. 50-57
