Native aspect ratio autoregressive image models

Abstract

A device includes a memory configured to store data corresponding to a trained transformer for next image patch prediction. The device also includes one or more processors coupled to the memory. The one or more processors are configured to obtain an input image and to generate input data that corresponds to a sequence of patches of the input image, the patches arranged to preserve an aspect ratio of the input image. The one or more processors are also configured to process the input data at the trained transformer to generate a transformer output that corresponds to one or more predicted next patches of the sequence.

Description

QUALCOMM Ref. No.2406633WO - 1 - NATIVE ASPECT RATIO AUTOREGRESSIVE IMAGE MODELS I. Cross-Reference to Related Applications

[0001] The present application claims the benefit of priority from the commonly owned Greece Patent Application No.20240100591, filed August 26, 2024, the contents of which are expressly incorporated herein by reference in their entirety. II. Field

[0002] The present disclosure is generally related to image models. III. Description of Related Art

[0003] In recent years, it has become clear that pre-training large transformer models on large datasets produces generally capable models, which can be fine-tuned to a wide variety of different tasks. In language modelling, these foundation models are created by training a large language model with billions of parameters using a next token prediction task and datasets containing hundreds of billions of tokens. Moreover, performance on this prediction task has been shown to scale well with data, computational resources, and model size.

[0004] By contrast, no pre-training task in computer vision has demonstrated similarly favorable scaling performance — in fact, it has been shown that the features learned by pre-training a model on reconstruction tasks such as Masked Autoencoders are fundamentally flawed when transferred to downstream perception tasks. Thus, there is a need for improved techniques of pre-training tasks for models in computer vision that provide enhanced accuracy and also demonstrate performance that scales with data, computer, and model size. IV. Summary According to a particular implementation of the techniques disclosed herein, a device includes a memory configured to store data corresponding to a trained transformer for next image patch prediction. The device also includes one or more processors coupled to the memory. The one or more processors are configured to obtain an input image andQUALCOMM Ref. No.2406633WO - 2 - to generate input data that corresponds to a sequence of patches of the input image, the patches arranged to preserve an aspect ratio of the input image. The one or more processors are also configured to process the input data at the trained transformer to generate a transformer output that corresponds to one or more predicted next patches of the sequence.

[0005] According to a particular implementation of the techniques disclosed herein, a method includes obtaining, at a device, an input image. The method includes generating, at the device, input data that corresponds to a sequence of patches of the input image, the patches arranged to preserve an aspect ratio of the input image. The method also includes processing, at a trained transformer for next image patch prediction at the device, the input data to generate a transformer output corresponding to one or more predicted next patches of the sequence.

[0006] According to a particular implementation of the techniques disclosed herein, a non-transitory computer-readable medium stores instructions that, when executed by one or more processors, cause the one or more processors to obtain an input image and to generate input data that corresponds to a sequence of patches of the input image, the patches arranged to preserve an aspect ratio of the input image. The instructions that, when executed by one or more processors, also cause the one or more processors to process, at a trained transformer for next image patch prediction, the input data to generate a transformer output corresponding to one or more predicted next patches of the sequence.

[0007] Other implementations, advantages, and features of the present disclosure will become apparent after review of the entire application, including the following sections: Brief Description of the Drawings, Detailed Description, and the Claims. V. Brief Description of the Drawings

[0008] FIG.1 is a block diagram illustrating an example of a system operable to perform image processing using a native aspect ratio at an autoregressive model, in accordance with some examples of the present disclosure.QUALCOMM Ref. No.2406633WO - 3 -

[0009] FIG.2 is a block diagram illustrating an example that includes operations that can be performed by the system of FIG.1, in accordance with some examples of the present disclosure.

[0010] FIG.3 is a diagram illustrating an example of operations that can be performed by the system of FIG.1, in accordance with some examples of the present disclosure.

[0011] FIG.4 is a diagram illustrating an example of components and operations that can be implemented in the system of FIG.1, in accordance with some examples of the present disclosure.

[0012] FIG.5 is a block diagram illustrating an example of components and operations that can be implemented in the system of FIG.1, in accordance with some examples of the present disclosure.

[0013] FIG.6 is a block diagram illustrating an example of components and operations that can be implemented in the system of FIG.1, in accordance with some examples of the present disclosure.

[0014] FIG.7 is a diagram illustrating an example of an image prediction of the system of FIG.1, in accordance with some examples of the present disclosure.

[0015] FIG.8 is a diagram illustrating an example of operations that can be performed by the system of FIG.1, in accordance with some examples of the present disclosure.

[0016] FIG.9 is a diagram illustrating a graph corresponding to a performance characteristic that may be exhibited by the system of FIG.1, in accordance with some examples of the present disclosure.

[0017] FIG.10 is a diagram illustrating a graph corresponding to a performance characteristic that may be exhibited by the system of FIG.1, in accordance with some examples of the present disclosure.QUALCOMM Ref. No.2406633WO - 4 -

[0018] FIG.11 is a diagram illustrating a graph corresponding to a performance characteristic that may be exhibited by the system of FIG.1, in accordance with some examples of the present disclosure.

[0019] FIG.12 is a diagram illustrating an example of one or more components of the system of FIG.1, in accordance with some examples of the present disclosure.

[0020] FIG.13 is a diagram illustrating an example of an integrated circuit operable to perform image processing using a native aspect ratio at an autoregressive model, in accordance with some examples of the present disclosure.

[0021] FIG.14 is a diagram of an example of a portable electronic device operable to perform image processing using a native aspect ratio at an autoregressive model, in accordance with some examples of the present disclosure.

[0022] FIG.15 is a diagram of an example of a camera operable to perform image processing using a native aspect ratio at an autoregressive model, in accordance with some examples of the present disclosure.

[0023] FIG.16 is a diagram of an example of a wearable electronic device operable to perform image processing using a native aspect ratio at an autoregressive model, in accordance with some examples of the present disclosure.

[0024] FIG.17 is a diagram of an example of an extended reality device, such as augmented reality glasses, operable to perform image processing using a native aspect ratio at an autoregressive model, in accordance with some examples of the present disclosure.

[0025] FIG.18 is a diagram of an example of a headset, such as a virtual reality, mixed reality, or augmented reality headset, operable to perform image processing using a native aspect ratio at an autoregressive model, in accordance with some examples of the present disclosure.QUALCOMM Ref. No.2406633WO - 5 -

[0026] FIG.19 is a diagram of an example of a voice-controlled speaker system operable to perform image processing using a native aspect ratio at an autoregressive model, in accordance with some examples of the present disclosure.

[0027] FIG.20 is a diagram of a first example of a vehicle operable to perform image processing using a native aspect ratio at an autoregressive model, in accordance with some examples of the present disclosure.

[0028] FIG.21 is a diagram of a second example of a vehicle operable to perform image processing using a native aspect ratio at an autoregressive model, in accordance with some examples of the present disclosure.

[0029] FIG.22 is a diagram of a particular example of a method of performing image processing using a native aspect ratio at an autoregressive model, in accordance with some examples of the present disclosure.

[0030] FIG.23 is a block diagram of a particular illustrative example of a device that is operable to perform image processing using a native aspect ratio at an autoregressive model, in accordance with some examples of the present disclosure. VI. Detailed Description

[0031] Systems and methods to perform image processing using a native aspect ratio at an autoregressive model are disclosed. While vision transformers are able to solve a wide variety of computer vision tasks, no pre-training method has yet demonstrated the same scaling performance as observed in language models. Autoregressive models show promising results for computer vision tasks, but so far they have been trained using images that are cropped and / or transformed into square images, which distorts or destroys information present in the input images. By maintaining the native aspect ratio (NAR) of input images, the disclosed techniques preserve the original spatial context of the input images, therefore enhancing the model’s ability to interpret visual information. Thus, a Native Aspect Ratio Autoregressive Image Model (NARAIM) - a vision model pre-trained with an autoregressive objective that uses images in their native aspect ratios, is presented.QUALCOMM Ref. No.2406633WO - 6 -

[0032] According to an aspect, the disclosed techniques relate to using the same objective – autoregressive next-token prediction – that is used to train large language models (LLMs) to train a vision transformer (ViT). The autoregressive objective outperforms the masking objective used with Masked Autoencoders for reconstruction tasks and demonstrates similar scaling performance as LLMs.

[0033] However, when pre-processing images for vision tasks, images are generally processed with an operation such as RandomResizedCrop, where images are first randomly cropped, and then resized to a fixed resolution. This pre-processing induces strong regularisation but comes at the cost of distorting image information. The present techniques combine the favourable scaling laws of the next-token prediction task with the advantages resulting from keeping images in their NAR. According to an aspect, to keep the regularisation benefits of the RandomResizedCrop, instead of using the entire image, a random crop of the input image is used and processed to generate a grid of patches of the random crop of the input image. Henceforth, the random crop taken from an image is referred to as an input.

[0034] The present techniques include an autoregressive approach for pre-training vision models using NAR inputs. Using NAR inputs improves the mean squared error (MSE) on the next-token prediction task, as well as the accuracy on the downstream classification task, while using no additional compute (e.g., the number of tokens fed to the transformer remains the same). Moreover, the disclosed techniques are more time-efficient when adapted to a classification task: for a fixed level of classification accuracy, fine-tuning a model pre- trained in square aspect ratio (SAR) takes more iterations than fine-tuning a model pre- trained in NAR.

[0035] Using NAR can include modification from conventional positional encodings, such as when as the transformer is not configured to determine a patch's location in the input solely from the patch's one-dimensional (1D) index. Instead, the disclosed techniques introduce use of 2D positional encodings, which may be implemented as absolute embeddings or fractional embeddings.

[0036] By pre-training an autoregressive image model on inputs in their NAR, the present techniques outperform the same image models, but trained on SAR inputs, on aQUALCOMM Ref. No.2406633WO - 7 - downstream classification task. Thus, the present techniques provide the advantage of enhanced performance for image-related processing, with reduced error, enhanced accuracy on downstream tasks, and scaling with data, compute, and model size in a similar manner as LLMs. In addition, models that are pre-trained on NAR inputs are more efficiently trained using fewer iterations during fine-tuning. As a result, the performance of a device implementing one or more of the disclosed techniques is improved.

[0037] Particular aspects of the present disclosure are described below with reference to the drawings. In the description, common features are designated by common reference numbers. As used herein, various terminology is used for the purpose of describing particular implementations only and is not intended to be limiting of implementations. For example, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. Further, some features described herein are singular in some implementations and plural in other implementations. To illustrate, FIG.1 depicts a device 102 including one or more processors (“processor(s)” 116 of FIG.1), which indicates that in some implementations the device 102 includes a single processor 116 and in other implementations the device 102 includes multiple processors 116. For ease of reference herein, such features are generally introduced as “one or more” features and are subsequently referred to in the singular or optional plural (as indicated by “(s)” in the name of the feature) unless aspects related to multiple of the features are being described.

[0038] In some drawings, multiple instances of a particular type of feature are used. Although these features are physically and / or logically distinct, the same reference number is used for each, and the different instances are distinguished by addition of a letter to the reference number. When the features as a group or a type are referred to herein, e.g., when no particular one of the features is being referenced, the reference number is used without a distinguishing letter. However, when one particular feature of multiple features of the same type is referred to herein, the reference number is used with the distinguishing letter. For example, referring to FIG.6, multiple patches 614 are illustrated and associated with reference numbers 614A and 614B. When referring to a particular one of these patches, such as a patch 614A, the distinguishing letter “A” isQUALCOMM Ref. No.2406633WO - 8 - used. However, when referring to any arbitrary one of these patches or to these patches as a group, the reference number 614 is used without a distinguishing letter.

[0039] As used herein, the terms “comprise,” “comprises,” and “comprising” may be used interchangeably with “include,” “includes,” or “including.” Additionally, it will be understood that the term “wherein” may be used interchangeably with “where.” As used herein, “exemplary” may indicate an example, an implementation, and / or an aspect, and should not be construed as limiting or as indicating a preference or a preferred implementation. As used herein, an ordinal term (e.g., “first,” “second,” “third,” etc.) used to modify an element, such as a structure, a component, an operation, etc., does not by itself indicate any priority or order of the element with respect to another element, but rather merely distinguishes the element from another element having a same name (but for use of the ordinal term). As used herein, the term “set” refers to one or more of a particular element, and the term “plurality” refers to multiple (e.g., two or more) of a particular element.

[0040] As used herein, “coupled” may include “communicatively coupled,” “electrically coupled,” or “physically coupled,” and may also (or alternatively) include any combinations thereof. Two devices (or components) may be coupled (e.g., communicatively coupled, electrically coupled, or physically coupled) directly or indirectly via one or more other devices, components, wires, buses, networks (e.g., a wired network, a wireless network, or a combination thereof), etc. Two devices (or components) that are electrically coupled may be included in the same device or in different devices and may be connected via electronics, one or more connectors, or inductive coupling, as illustrative, non-limiting examples. In some implementations, two devices (or components) that are communicatively coupled, such as in electrical communication, may send and receive signals (e.g., digital signals or analog signals) directly or indirectly, via one or more wires, buses, networks, etc. As used herein, “directly coupled” may include two devices that are coupled (e.g., communicatively coupled, electrically coupled, or physically coupled) without intervening components.

[0041] In the present disclosure, terms such as “obtaining,” “determining,” “calculating,” “estimating,” “shifting,” “adjusting,” etc. may be used to describe howQUALCOMM Ref. No.2406633WO - 9 - one or more operations are performed. It should be noted that such terms are not to be construed as limiting and other techniques may be utilized to perform similar operations. Additionally, as referred to herein, “obtaining,” “generating,” “calculating,” “estimating,” “using,” “selecting,” “accessing,” and “determining” may be used interchangeably. For example, “obtaining,” “generating,” “calculating,” “estimating,” or “determining” a parameter (or a signal) may refer to actively generating, estimating, calculating, or determining the parameter (or the signal) or may refer to using, selecting, retrieving, receiving, or accessing the parameter (or signal) that is already generated, such as by another component or device.

[0042] As used herein, the term “machine learning” should be understood to have any of its usual and customary meanings within the fields of computers science and data science, such meanings including, for example, processes or techniques by which one or more computers can learn to perform some operation or function without being explicitly programmed to do so. As a typical example, machine learning can be used to enable one or more computers to analyze data to identify patterns in data and generate a result based on the analysis. For certain types of machine learning, the results that are generated include data that indicates an underlying structure or pattern of the data itself. Such techniques, for example, include so called “clustering” techniques, which identify clusters (e.g., groupings of data elements of the data).

[0043] For certain types of machine learning, the results that are generated include a data model (also referred to as a “machine-learning model” or simply a “model”). Typically, a model is generated using a first data set to facilitate analysis of a second data set. For example, a first portion of a large body of data may be used to generate a model that can be used to analyze the remaining portion of the large body of data. As another example, a set of historical data can be used to generate a model that can be used to analyze future data.

[0044] Since a model can be used to evaluate a set of data that is distinct from the data used to generate the model, the model can be viewed as a type of software (e.g., instructions, parameters, or both) that is automatically generated by the computer(s) during the machine learning process. As such, the model can be portable (e.g., can beQUALCOMM Ref. No.2406633WO - 10 - generated at a first computer, and subsequently moved to a second computer for further training, for use, or both). Additionally, a model can be used in combination with one or more other models to perform a desired analysis. To illustrate, first data can be provided as input to a first model to generate first model output data, which can be provided (alone, with the first data, or with other data) as input to a second model to generate second model output data indicating a result of a desired analysis. Depending on the analysis and data involved, different combinations of models may be used to generate such results. In some examples, multiple models may provide model output that is input to a single model. In some examples, a single model provides model output to multiple models as input.

[0045] Examples of machine-learning models include, without limitation, perceptrons, neural networks, support vector machines, regression models, decision trees, Bayesian models, Boltzmann machines, adaptive neuro-fuzzy inference systems, as well as combinations, ensembles and variants of these and other types of models. Variants of neural networks include, for example and without limitation, prototypical networks, autoencoders, transformers, self-attention networks, convolutional neural networks, deep neural networks, deep belief networks, etc. Variants of decision trees include, for example and without limitation, random forests, boosted decision trees, etc.

[0046] Since machine-learning models are generated by computer(s) based on input data, machine-learning models can be discussed in terms of at least two distinct time windows – a creation / training phase and a runtime phase. During the creation / training phase, a model is created, trained, adapted, validated, or otherwise configured by the computer based on the input data (which in the creation / training phase, is generally referred to as “training data”). Note that the trained model corresponds to software that has been generated and / or refined during the creation / training phase to perform particular operations, such as classification, prediction, encoding, or other data analysis or data synthesis operations. During the runtime phase (or “inference” phase), the model is used to analyze input data to generate model output. The content of the model output depends on the type of model. For example, a model can be trained to perform classification tasks or regression tasks, as non-limiting examples. In some implementations, a model may be continuously, periodically, or occasionally updated,QUALCOMM Ref. No.2406633WO - 11 - in which case training time and runtime may be interleaved or one version of the model can be used for inference while a copy is updated, after which the updated copy may be deployed for inference.

[0047] In some implementations, a previously generated model is trained (or re- trained) using a machine-learning technique. In this context, “training” refers to adapting the model or parameters of the model to a particular data set. Unless otherwise clear from the specific context, the term “training” as used herein includes “re-training” or refining a model for a specific data set. For example, training may include so called “transfer learning.” In transfer learning a base model may be trained using a generic or typical data set, and the base model may be subsequently refined (e.g., re-trained or further trained) using a more specific data set.

[0048] A data set used during training is referred to as a “training data set” or simply “training data.” The data set may be labeled or unlabeled. “Labeled data” refers to data that has been assigned a categorical label indicating a group or category with which the data is associated, and “unlabeled data” refers to data that is not labeled. Typically, “supervised machine-learning processes” use labeled data to train a machine- learning model, and “unsupervised machine-learning processes” use unlabeled data to train a machine-learning model; however, it should be understood that a label associated with data is itself merely another data element that can be used in any appropriate machine-learning process. To illustrate, many clustering operations can operate using unlabeled data; however, such a clustering operation can use labeled data by ignoring labels assigned to data or by treating the labels the same as other data elements.

[0049] Training a model based on a training data set generally involves changing parameters of the model with a goal of causing the output of the model to have particular characteristics based on data input to the model. To distinguish from model generation operations, model training may be referred to herein as optimization or optimization training. In this context, “optimization” refers to improving a metric, and does not mean finding an ideal (e.g., global maximum or global minimum) value of the metric. Examples of optimization trainers include, without limitation, backpropagation trainers, derivative free optimizers (DFOs), and extreme learning machines (ELMs). AsQUALCOMM Ref. No.2406633WO - 12 - one example of training a model, during supervised training of a neural network, an input data sample is associated with a label. When the input data sample is provided to the model, the model generates output data, which is compared to the label associated with the input data sample to generate an error value. Parameters of the model are modified in an attempt to reduce (e.g., optimize) the error value. As another example of training a model, during unsupervised training of an autoencoder, a data sample is provided as input to the autoencoder, and the autoencoder reduces the dimensionality of the data sample (which is a lossy operation) and attempts to reconstruct the data sample as output data. In this example, the output data is compared to the input data sample to generate a reconstruction loss, and parameters of the autoencoder are modified in an attempt to reduce (e.g., optimize) the reconstruction loss.

[0050] Referring to FIG.1, a particular illustrative example of a system 100 is depicted that includes a device 102 that is configured to perform image processing using a native aspect ratio at an autoregressive model. For example, the device 102 is configured to process input image(s) 122 using a machine learning (ML) model 140 that includes a trained transformer 142 configured to perform next image patch prediction for patches of the input image(s) 122 that maintain the native aspect ratio of the respective input image. Use of the trained transformer 142 with sequences of patches that preserve the native aspect ratio improves performance as compared to scaling the input images to have a square aspect ratio.

[0051] Optionally, the device 102 includes, or is coupled to, one or more image sensors 104. The image sensor 104 is configured to generate image data 105 that, in some embodiments, corresponds to the input image(s) 122. In a particular embodiment, the image sensor 104 corresponds to or is incorporated into a camera, such as a still image camera, a video camera, a stereo camera, a thermal imaging camera, one or more other types of camera, or a combination thereof. According to an aspect, the image data 105 includes data (e.g., pixel values) of individual images, video data, or a combination thereof.

[0052] The device 102 includes a memory 110 coupled to the one or more processors 116 and configured to store instructions 112 and the input image(s) 122, suchQUALCOMM Ref. No.2406633WO - 13 - as individual images or data corresponding to images included in video data (e.g., video frames). For example, the memory 110 may store a first image 124 and a second image 126 as part of the input image(s) 122 to be processed at the ML model 140, as described in further detail below. The memory 110 may also store data (e.g., parameters, such as weights and biases) associated with one or more ML models, such as the ML model 140, that may be implemented at the one or more processors 116. In a particular implementation, the memory 110 corresponds to a dynamic random access memory (DRAM) of a double data rate (DDR) memory subsystem.

[0053] The one or more processors 116 are configured to execute the instructions 112 to perform operations associated with the ML model 140. In various implementations, some or all of the functionality associated with ML model 140 is performed via execution of the instructions 112 by the one or more processors 116, performed by processing circuitry of the one or more processors 116 in a hardware implementation, or a combination thereof.

[0054] The one or more processors 116 may include an input image source 120 coupled to the ML model 140 and configured to provide the input image(s) 122 for processing by the ML model 140. For example, the input image source 120 may correspond to the image sensor 104, a portion of one or more of media files (e.g., a media file including the input images 124, 126 that is retrieved from the memory 110), one or more other sources of input images, such as from a remote media server, or a combination thereof.

[0055] The one or more processors 116 are configured to obtain an input image from the input image source 120, such as the first image 124, and to generate input data 136 that corresponds to a sequence of patches 138 of the input image 124, where the patches are arranged to preserve an aspect ratio of the input image. For example, the one or more processors 116 include a native aspect ratio (NAR)-preserving patch generator 130 that is configured to process the first image 124 by dividing the first image 124 (or a scaled version of the first image 124 that preserves the aspect ratio of the first image 124) into a grid of equally-sized patches. According to an aspect, the NAR-preserving patch generator 130 outputs the patches in a raster scan order (e.g., from left to rightQUALCOMM Ref. No.2406633WO - 14 - along each row of the grid, from the top row to the bottom row) to generate the sequence of patches 138. An example of operations that may be performed by the NAR- preserving patch generator 130 is described in further detail with reference to FIG.2.

[0056] According to an embodiment, the one or more processors 116 are further configured to include, in the input data 136, position information based on a horizontal image position and a vertical image position of each patch in the sequence of patches 138. For example, the position information may represent, or may be generated based on, a row index and column index of the patch, coordinates of one or more pixels associated with the patch, or other values that represent or are derived from the row / column indices or coordinates. In some examples, the position information may be added at a linear projection network that is coupled to an input of the trained transformer 142, such as described further with reference to FIG.5.

[0057] The one or more processors 116 include the ML model 140, which is configured to process the input data 136 to generate model output data 152. The ML model 140 includes the trained transformer 142 for next image patch prediction and a prediction head 146. According to an aspect, the trained transformer 142 is configured to process the input data 136 to generate a transformer output 144 that corresponds to one or more predicted next patches of the sequence 138. In an example, the trained transformer 142 corresponds to an autoregressive vision transformer that has been pre- trained to generate, based on a sequence of one or more input patches, a transformer output 144 that contains sufficient information to accurately predict the next patch of the sequence, such as described further with respect FIG.6.

[0058] The prediction head 146 is configured to process the transformer output 144 to generate the model output data 152. In an example, the prediction head 146 corresponds to a trained classifier that is configured to output a classification of the input image based on the transformer output 144. In another example, the prediction head 146 is configured to perform image segmentation to generate a segmentation map as the model output data 152. In some examples, the prediction head 146 can correspond to image generation, such as included in a patch-wise diffusion model for generating realistic images. In other examples, the ML model 140 includes a multi-QUALCOMM Ref. No.2406633WO - 15 - modal model, such as a language model integrated with or coupled to the trained transformer 142. To illustrate, the prediction head 146 can include a large language model that inputs the transformer output 144 and generates a textual caption or description of the input image based on the transformer output 144, and the textual caption or description is included in the model output data 152. It should be understood that the above examples are provided for purposes of illustration and should not be considered limiting; in other examples, the prediction head 146 is configured to perform various other operations to generate other types of model output data 152.

[0059] Although in some embodiments the ML model 140 (e.g., the trained transformer 142, the prediction head 146, or both) may be trained at the device 102, in other embodiments the ML model 140 is not trained at the device 102. To illustrate, the ML model 140 may be trained at a remote device, such as a remote device 198, and the trained ML model 140 may be transmitted to the device 102 and stored in the memory 110. Aspects of training the ML model 140 are described in further detail with reference to FIGS.3, 4, and 6.

[0060] The device 102 optionally includes or is coupled to a display device 106. The display device 106 is configured to display output data 107 corresponding to, or based on, the model output data 152, for viewing by a user of the device 102. For example, in some embodiments in which the one or more processors 116 generate a 3D representation of an object or scene based on the image data 105, the one or more processors 116 may generate the output data 107 based on the 3D representation, such as in an extended reality application.

[0061] The device 102 optionally includes a modem 118 that is coupled to the one or more processors 116 and configured to enable communication with one or more other devices, such as via one or more wireless networks. According to some aspects, the modem 118 is configured to receive the image data 105, the input image(s) 122, or both, from a second device, such as image data (e.g., included in video data) that is streamed via a wireless transmission 194 from a remote device, such as the remote device 198 (e.g., a remote server) for processing at the device 102. According to some aspects, the modem 118 is configured to send data corresponding to the model output data 152 to aQUALCOMM Ref. No.2406633WO - 16 - second device, such as image data that is streamed via the wireless transmission 194 to a remote device 198 (e.g., a remote server or user device) for storage or playback.

[0062] A technical advantage of using the ML model 140 is that, as compared to conventional techniques in which scaling is employed to distort the aspect ratio of input images to generate SAR input images, the generation and processing of NAR- preserving patches by the NAR-preserving patch generator 130 and the ML model 140, respectively, provides enhanced accuracy for downstream tasks (e.g., for generation of the model output data 152), thus improving operation of the device 102.

[0063] According to some aspects, the one or more processors 116 are integrated in an integrated circuit, such as illustrated in FIG.13. According to some aspects, the one or more processors 116 are integrated in at least one of a mobile phone or a tablet computer device, such as illustrated in FIG.14, a camera device, such as illustrated in FIG.15, or a wearable electronic device, such as illustrated in FIG.16. According to some aspects, the one or more processors 116 are integrated in a headset device that includes a display and that is configured, when worn by a user, to display an output image based on an output of the ML model 140, such as illustrated in FIG.17 and FIG. 18. According to some aspects, the one or more processors 116 are integrated in a voice- controlled speaker system, such as illustrated in FIG.19. According to some aspects, the one or more processors 116 are integrated in a vehicle that also includes one or more cameras configured to capture image data corresponding to the input image(s) 122, such as illustrated in FIG.20 and FIG.21.

[0064] It should be understood that one or more aspects of the device 102 may have been omitted from the above description for clarity of explanation. For example, although in some embodiments the input image(s) 122 matches the image data 105, in other embodiments the input image(s) 122 may be the result of additional processing that is performed on the image data 105. Such additional processing can include cropping, zooming, tone mapping, color enhancement, upscaling, or downscaling, as illustrative, non-limiting examples.

[0065] FIG.2 depicts an example 200 that includes operations that may be implemented in the device 102 of FIG.1, according to some examples of the presentQUALCOMM Ref. No.2406633WO - 17 - disclosure. In particular, the example 200 illustrates operations 204 that may be implemented in an image pre-processing pipeline of the NAR-preserving patch generator 130, for comparison to operations 202 that may be associated with an image pre-processing pipeline associated with SAR patch generation.

[0066] In both image pre-processing pipelines, an input image (e.g., the first image 124 of FIG.1) is cropped to generate an image 210 that has a first aspect ratio 212 (e.g., a ratio of a first number of pixels in a vertical direction (height) to a second number of pixels in a horizontal direction (width)).

[0067] For SAR patch generation, the operations 202 include rescaling the image 210 to generate a scaled image 220 having a 224 × 224 square resolution, e.g., a second aspect ratio 222 of approximately 1. The operations 202 also include generating a 16 x 16 grid of patches 230 including a representative patch 232, where each patch corresponds to a 14 pixel x 14 pixel portion of the scaled image 220.

[0068] In contrast, the operations 204 associated with NAR processing include rescaling the image 210 such that the number of pixels in the resulting scaled image 240 is approximately equal to 2242(= 50,176) and the aspect ratio is preserved. To illustrate, the scaled image 240 has a height of 173 pixels and a width of 288 pixels, such that the number of pixels in the scaled image 240 is 173 x 288 (= 49,824 pixels), and the scaled image 240 has a third aspect ratio 242 that substantially matches the first aspect ratio 212. The operations 204 also include generating a 12 x 20 grid of patches 250 including a representative patch 252, where each patch corresponds to a 14 pixel x 14 pixel portion of the scaled image 240. Because patches having size 14 pixels × 14 pixels are used, at most 13 pixels will be cropped off (e.g., cropped off the right / bottom side) if the height and / or width of the scaled image 240 is not evenly divisible by 14.

[0069] FIG.3 depicts an example 300 of operations that can be performed by the device 102 of FIG.1, according to some examples of the present disclosure. In particular, the operations may be implemented to pre-train a transformer, such as to generate the trained transformer 142 of FIG.1, and to train a prediction head, such as the prediction head 146, to perform a downstream task using the output of the pre- trained transformer.QUALCOMM Ref. No.2406633WO - 18 -

[0070] In the example 300, a setup operation 302 includes obtaining a dataset of images. The images may be labeled or unlabeled. A pre-training operation 304 includes training a prefix causal transformer on the next-token prediction task while maintaining the native aspect ratio of images. For example, a prefix causal ViT may be pre-trained to predict the next patch of a sequence of patches from an image after sequentially inputting one or more prior patches of the sequence, while persevering the image’s native aspect ratio. To illustrate, the image patches may be generated in accordance with the operations 204 of FIG.2.

[0071] A downstream operation 306 includes obtaining a dataset with labelled images for the task of interest and using output representations of the pre-trained transformer to train the prediction head for the downstream task. The labelled dataset used to train the prediction head, to generate predictions based on the output of the pre- trained transformer, can be smaller than the dataset that was used to pre-train the transformer.

[0072] According to an aspect, a model (e.g., the trained transformer 142 of the ML model 140) is generated via pre-training with an autoregressive objective: given an ଷ×ு×^ input ^^ ∈ ℝ , the input is split into flattened patches [x1, x2, ..., xN], using a raster(row-major) ordering, where N is the number of patches. The splitting results in ^^^^ ∈ℝଷ^మ, with P being the height and width of a patch. Hence, ^^ = ^ு^^ ∙The modeltakes the patches, linearly projects them, adds positional embeddings, and then passes them through a ViT. Finally, the features created by the ViT are passed to either a pre- training head that aims to predict patch x given all the previous patches x in rasteri+1 k (row-major) order (with 1 ≤ k ≤ i), or a classification head that aims to predict the class corresponding to X, such as described in further detail with reference to FIG.4. A visual example of the output of the pre-training head is described further herein with reference to FIG 7.

[0073] Two different loss functions can be used for the pre-training head: with andwithout patch normalisation. Let ^^^ ∈ ℝଷ^మ th^ be the i patch, where P is the height andwidth of a patch. Then, with normalisation, the ith prediction target ∈ ℝଷ^మQUALCOMM Ref. No.2406633WO - 19 -

[0075] where the sum of a vector and a scalar is to be mapped over the components of the vector. Here, μ(xi) and σ(xi) are defined to be the mean and standard deviation of the sequence of numbers contained in xi. More precisely, let xi,j ∈ R be the jthsubpixelin patch xi. Then, ^^:ℝଷ^మ→ ℝ is defined as

[0079] Without normalisation, the prediction target is simply

[0080] yi = xi+1.

[0081] The loss function is the mean squared error (MSE) between the predicted and ground-truth input: the squared differences per subpixel can then be summed and divided by the total number of subpixels in the input. Let N be the number of patches in the input. Then, there are N − 1 predicted patches (x1is not predicted), so a total of (N −1)(3P2) subpixels in the input:

[0083] For fine-tuning on the classification task, a standard cross entropy loss function can be used.

[0084] FIG.4 depicts an example 400 of components and operations that can be implemented in the system 100 of FIG.1, in accordance with some examples of the present disclosure. In particular, the example 400 graphically depicts operations that can be performed in accordance with the operations of FIG.3.QUALCOMM Ref. No.2406633WO - 20 -

[0085] As shown in FIG.4, an unlabeled image 402, such as from the dataset of images of the setup operation 302, is processed to generate a sequence of patches (sequentially numbered 1-9 for illustrative purposes) in accordance with the NAR- preserving operations 204 of FIG.2. A first configuration 404 of a ViT model illustrates a prefix causal transformer 442A coupled to a pre-training head 446A, such as a multi-level perceptron (MLP) for pre-training (e.g., the pre-training operation 304), and a second configuration 406 of the ViT model illustrates a pre-trained transformer 442B (e.g., the trained transformer 142) coupled to a classification head 446B for downstream training (e.g., the downstream operation 306).

[0086] The illustrated model can include a projection (such as described further with reference to FIG.5), positional embedding (such as described further with reference to FIG.5), and the transformer 442, followed by one of the two prediction heads 446, chosen depending on the task. During pre-training, the pre-training head 446A is used, illustrated in the first configuration 404 as an MLP that takes the features output by the transformer 442A to predict the next patch given all previous patches. To evaluate the quality of the features, the base model is frozen (e.g., the model parameters (e.g., weights) are held constant), and the pre-training head 446A can be switched out with the classification head 446B — an attentive probe that uses the transformer’s output features to predict the class. Although an attentive probe is illustrated, other configurations may be used, such as a linear probe. For purposes of comparing performance using SAR inputs to performance using NAR inputs, the input to the classifier is in the same format as the input during pre-training: a model that was pre-trained on SAR inputs is used to classify SAR inputs, and a model pre-trained with NAR inputs is used to classify NAR inputs.

[0087] Adapting a model trained with causal attention to down-stream tasks may cause issues: due to the attention mask, the model only learns to create representations using causal attention. This means that for a token xi, only the information from tokens {x1, ..., xi−1} will be incorporated. However, in some embodiments, every token should be able to attend to every other token in downstream tasks.

[0088] Prefix causal attention addresses this issue by selecting a random integer n between 1 and N − 1, where N is the number of tokens in an input. Then, for tokensQUALCOMM Ref. No.2406633WO - 21 - {x1, ..., xn} the causal attention mask is dropped, and every token in this set can attend to every other token in the set. For the remaining tokens, {xn+1, ..., xN}, the standard causal attention mask is used. Predictions for tokens {x1, ..., xn} are not included in the loss calculation as the tokens could be trivially predicted using the identity function. An example of prefix causal attention is described in further detail with reference to FIG.8.

[0089] Positional embeddings play a significant role in helping the model understand the spatial structure of inputs. Various types of positional embedding may be used, such as absolute position embeddings and fractional position embeddings.

[0090] To calculate the fractional position embedding for patch xi, let hi, wibe the vertical and horizontal indices of the patch xiin the input, with H, W the total number of vertical and horizontal patches in the input. Then, the fractional position embedding for the patch is f (hi / H) + g(wi / W ), where f and g are learnable one-layer perceptrons in some embodiments.

[0091] Regarding the absolute position embeddings, let hi, wiagain be the vertical and horizontal indices, and let ϕ be the function:

[0092] ϕ(pos, 2i) = sin(pos / 100002i / d),

[0093] and

[0094] ϕ(pos, 2i + 1) = cos(pos / 100002i / d),

[0295] where d corresponds to the feature dimension: d = ½(dmodel). The complete absolute position embedding is then:

[0096] ϕ(hi, 1 : d) ⊕ ϕ(wi, 1 : d) ,

[0097] where ⊕ denotes concatenation.

[0098] Although absolute position embeddings are straightforward to implement and integrate into different ViTs, they might not be an optimal option for cases in which images have different resolutions: if, during inference, the model would receive anQUALCOMM Ref. No.2406633WO - 22 - image that is either wider or taller than every image in the training set, the positional embedding for that image would be out of domain for the model.

[0099] Since NARAIM receives inputs in their native aspect ratio (and, consequently, different resolutions), fractional positional embeddings may be expected to provide improved results under some circumstances.

[0100] FIG.5 depicts an example 500 of components and operations that may be implemented in the system 100 of FIG.1. In particular, the example 500 illustrates a linear projection network 502 that may be included in the one or more processors 116 and configured to receive a patch 510, such as from the NAR-preserving patch generator 130, and to generate input data 560 (e.g., the input data 136) associated with the patch 510. The linear projection network 502 may be coupled to an input of the trained transformer 142 to provide the input data 560 to the trained transformer 142.

[0101] The linear projection network 502 includes a linear projector 520 (e.g., a single neural network layer) configured to receive the patch 510 (e.g., a 14x14 set of pixel values) and to output a patch embedding 530. A position adder 540 is configured to add position information 542 to the patch embedding 530 to include the position information 542 in the input data 560. The position information 542 may be based on a horizontal image position (xPos) 544 and a vertical image position (yPos) 546 of each patch and, in some examples, include a position embedding that is based on the image positions. In a particular example, the position information 542 corresponds to an absolute position embedding. In another example, the position information 542 corresponds to a fractional position embedding.

[0102] FIG.6 depicts an example 600 of components and operations that may be implemented in the system 100 of FIG.1. In particular, the example 600 illustrates a next-patch prediction training process 602 that may be performed by the one or more processors 116 to pre-train a vision transformer 630 to generate the trained transformer 142, and optionally to train a pre-training head 640 to generate the prediction head 146.

[0103] The operations of the next-patch prediction training process 602 include obtaining training data 604 that includes, for a first training image 610 (“Training ImageQUALCOMM Ref. No.2406633WO - 23 - 1”) of multiple training images, a respective sequence 612 of patches 614 of the training image 610. The patches 614 may be arranged to preserve an aspect ratio of the first training image 610. To illustrate, the training data 604 includes a first sequence 612 of patches 614 of a first training image 610, and the patches 614 include a first patch 614A, a second patch 614B, and one or more additional patches, arranged to preserve an aspect ratio of the first training image 610. The training data 604 also includes one or more sequences of patches for one or more additional training images, including a second sequence 622 of patches 624 of a second training image 620 (“Training Image N”), and the patches 624 include a first patch 624A, a second patch 624B, and one or more additional patches, arranged to preserve an aspect ratio of the second training image 620.

[0104] In a particular example, the training data 604 is generated at the device 102 by the NAR-preserving patch generator 130 and includes embedded position data of the patches, such as generated via the linear projection network 502 of FIG.5. The one or more processors 116 of FIG.1 may be configured to, for each training image of the multiple training images, partition and scale the training image to generate multiple equally-sized patches for the training image that preserve the aspect ratio of the training image, such as the grid of patches 250 of FIG.2 that preserve the aspect ratio 212 of the image 210.

[0105] The operations also include performing the next-patch prediction training process 602 on a vision transformer 630, using the training data 604, to generate the trained transformer 142. According to an aspect, during the next-patch prediction training process 602, the one or more processors 116 are configured to, for each training image of the multiple training images: process an output 632 of the vision transformer 630 at a pre-training head 640 to generate predicted next patches 650 in a respective sequence of patches; and update the vision transformer 630 based on an accuracy 660 of the predicted next patches 650. For example, the vision transformer 630 may process the first patch 614A of the first sequence 612 to generate an output 632 that is converted by the pre-training head 640 into a prediction of the second patch 614B, which is compared to the second patch 614B to determine the accuracy 660 based on a loss function that isQUALCOMM Ref. No.2406633WO - 24 - used to update the vision transformer 630 (and optionally, to update the pre-training head 640).

[0106] Although the next-patch prediction training process 602 is described as being performed by the one or more processors 116 of the device 102, it should be understood that, in other embodiments, the next-patch prediction training process 602 is instead performed at another device, such as the remote device 198 of FIG.1 (e.g., one or more servers of a server farm), that may have greater processing capacity than the device 102 (e.g., a mobile or wearable device), and the resulting trained vision transformer 630 may be sent to the device 102 for use as the trained transformer 142.

[0107] FIG.7 depicts an example 700 of an input image 702 that may be input to the ML model 140 and a corresponding output image 704 that may be generated using next patch predictions of the ML model 140, such as the predicted next patches 650 of FIG.6 that correspond to the first training image 610 upon completion of the next-patch prediction training process 602, or the output of the pre-training head (MLP) 446A of FIG.4, as non-limiting examples. In the example 700, each predicted patch xiin the output image 704 is predicted based on the previous patches x1, …, xi-1.

[0108] FIG.8 depicts a simplified graphical example 800 of prefix causal attention during pre-training in a first diagram 802, and during fine-tuning in a second diagram 804, according to an aspect.

[0109] For pre-training, a prefix length n is uniformly sampled during pre-training (e.g., n = 3 as illustrated in the first diagram 802). The attention for the first n patches is set to be bidirectional and no loss will be computed for them. The rest of the patches adopt a causal mask and their loss is calculated. During fine-tuning to a downstream task (as illustrated in the second diagram 804), the mask is discarded. The right-most patches represent the padding, which are added for reasons explained below.

[0110] FIGS.9-11 illustrate results of various comparisons. For the comparisons, the ImageNet-1k dataset was used for both pre-training and evaluation of downstream adaptation. All input sequences are constrained to have the same length to be processed by the ViT, so each input sequence is padded to a common length. To avoid wasting aQUALCOMM Ref. No.2406633WO - 25 - lot of computational resources on padded tokens, the number of patches in each image is approximately the same. To this end, images are first resized such that their area is smaller than or equal to 2242, and then rounded to the nearest integer values. Next, a crop is taken from the top-left such that the height and width of the crop are some integer multiple of the patch size. A center crop could be taken instead of the top-left crop, but with the small patch size, this is unlikely to produce a significant difference. After cropping, the image is split into patches of 14 × 14 which are processed by the ViT and used to predict either the next patch or the image’s class.

[0111] A random horizontal flip is used for both pre-training and fine-tuning. In NAR, for both pre-training and fine-tuning, both not cropping or using a random crop with a scale sampled from U(0.4, 1) and an aspect ratio sampled from U(0.75, 1.33) were experimented with. Additionally, the constraint that the crop needs to contain at least 2242pixels was imposed. This ensured that every input had enough information to learn from. In SAR, the RandomResizedCrop like in AIM with scale and aspect ratio sampled as above was used.

[0112] To test the model described herein, a ViT-B model is used as backbone with 12 attention heads, an embedding dimension of 768, a total of 12 transformer layers, and MLP dimension of 3,072.

[0113] To provide a fair comparison between pre-training with NAR and SAR, both models are trained from scratch with the same objective. The MSEs of the pre-training objectives and the classification accuracies of the fine-tuned models are then compared, and the classification heads are fine-tuned for 50,000 iterations.

[0114] To put the attained classification accuracies in perspective, an attentive probe is also directly trained on image patches, and an attentive probe on the features output by a randomly initialised and untrained ViT-B.

[0115] As an initial confirmation that the model is learning useful features, the results of a fully fine-tuned (i.e., no frozen backbone) pre-trained model is compared to a model trained on classification from scratch, as illustrated in FIG.9. In FIG.9, theQUALCOMM Ref. No.2406633WO - 26 - vertical axis indicates accuracy (e.g., IN-1k Top-1). As illustrated, the pre-trained model performs much better than the model trained from scratch.

[0116] Next, the performance of a ViT trained using images in their native aspect ratio is compared with a ViT trained using images in square aspect ratio. The results indicate that using NAR improves over SAR for both the pre-training MSE as well as the classification accuracy. A ViT trained with SAR also takes more steps to attain the same accuracy as NAR, as illustrated in FIG.10. In FIG.10, validation accuracy over time of the attentive probe is depicted, for vision transformers trained with different strategies, with all models using absolute positional encodings. As the aim of these pre-trained models is to adapt them to downstream tasks, NAR reaches a given level of accuracy faster than SAR; this implies that NAR is more efficient to fine-tune than SAR.

[0117] Comparing this to attentive probes trained directly on image patches and on random transformers, it can be appreciated that the difference between SAR and NAR is substantial: going from random transformer features to a network trained with SAR improves accuracy by about 14 percentage points, and going from SAR to NAR improves accuracy by approximately 4 percentage points.

[0118] According to some examples, using fractional positional embeddings instead of absolute positional embeddings provides a moderate increase in performance. The reason for this might be that the learned parameters provide the network with some extra expressivity. As for the patch normalisation, removing it has been found to decrease the MSE of the pre-training task, but this is to be expected as the prediction task will be easier: without normalisation, the input values are between 0 and 1, and thus, the variance will always be less than 0.25 by Popoviciu’s inequality. Hence, it is expected that the errors will be smaller than after normalisation. For the fine-tuned classification accuracy, this decreases slightly compared to the baseline NAR, indicating that the learnt features are slightly worse.

[0119] Adding AutoAugment reduces overfitting, but does not seem to affect downstream accuracy. The model may improve if trained for longer, as it may need more time to converge than the baseline model. Another possibility is thatQUALCOMM Ref. No.2406633WO - 27 - AutoAugment distorts the information in the image too much, and that the network would benefit more from less harsh augmentation.

[0120] FIG.11 depicts an example of pre-training performance over time. The downstream models were fine-tuned for 50,000 iterations. The models were both pre- trained and fine-tuned in native aspect ratio, with absolute positional encodings.

[0121] The disclosed device and techniques demonstrate the application of NAR to generative pre-training of ViTs. The comparisons described above demonstrate that using NAR improves the performance of the pre-training head, improves the downstream classification accuracy, and improves the fine-tuning efficiency. Using fractional positional encodings was also found to provide a small performance boost over using absolute positional encodings. Taken together, these results are an indication that pre-training autoregressive image models in NAR results in significant improvement, resulting in more accurate device operation and reduced training times as compared to conventional techniques.

[0122] FIG.12 illustrates another example implementation in which the ML model 140 (which includes the trained transformer 142) corresponds to, or is integrated in, a multi-modal foundational model with end-to-end pre-training and native resolution. As illustrated, the ML model 140 receives multi-modal inputs, including linear projections generated by a linear projection network 1202 (e.g., the linear projection network 502 of FIG.5) of each image patch of a sequence of image patches 1210, which can be interspersed with a dedicated separator embedding (illustrated as “\n”) to distinguish between rows when the sequence of image patches is input, and also to distinguish between image patch inputs and textual inputs.

[0123] FIG.13 is a block diagram illustrating an example 1300 of the device 102 as an integrated circuit 1302 for performing image processing using a native aspect ratio at an autoregressive model. The integrated circuit 1302 includes the one or more processors 116, which include the ML model 140 (e.g., including the trained transformer 142 and the prediction head 146, and optionally including the NAR- preserving patch generator 130, the linear projection network 502, or both). The integrated circuit 1302 also includes a signal input 1304, such as a bus interface, toQUALCOMM Ref. No.2406633WO - 28 - enable input data 1305, such as the image data 105, the input image(s) 122, or the input data 136 to be received. The integrated circuit 1302 includes a signal output 1306, such as a bus interface, to enable outputting of output data 1307, such as the model output data 152 or the output data 107. Optionally, the integrated circuit 1302 also includes the memory 110, the image sensor 104, the input image source 120, the modem 118, a display engine, etc. The integrated circuit 1302 enables implementation of input data processing (e.g., image processing using a native aspect ratio at an autoregressive model) as a component in a system that performs image processing, such as depicted in FIG.1.

[0124] FIG.14 depicts an example 1400 in which the device 102 includes, corresponds to, or is integrated within a mobile device 1402, such as a phone or tablet, as illustrative, non-limiting examples. The mobile device 1402 includes a display screen 1404 and a camera 1412 (e.g., the image sensor 104). The ML model 140 is integrated in the mobile device 1402, such as in the integrated circuit 1302, which is illustrated using dashed lines to indicate internal components that are not generally visible to a user of the mobile device 1402. In a particular example, the ML model 140 operates to perform image processing using a native aspect ratio at an autoregressive model. For example, the mobile device 1402 may generate the image data 105 from the camera 1412, process the image data 105 using the ML model 140, and display the resulting output data 107 at the display screen 1404 and / or transmit the resulting output data 107 or the model output data 152 to another device, such as the remote device 198.

[0125] FIG.15 depicts an example 1500 in which the device 102 includes, corresponds to, or is integrated within a portable electronic device that corresponds to a camera device 1502. The camera device 1502 includes an image sensor 1512, such as the image sensor 104. The ML model 140 is integrated in the camera device 1502, such as in the integrated circuit 1302. In a particular example, the ML model 140 operates to perform image processing using a native aspect ratio at an autoregressive model. For example, the camera device 1502 may generate the image data 105 from the image sensor 1512, process the image data 105 using the ML model 140, and display the resulting output data 107 at a display screen of the camera device 1502, store the resulting output data 107 or the model output data 152 at a memory of the cameraQUALCOMM Ref. No.2406633WO - 29 - device 1502, and / or transmit the resulting output data 107 or the model output data 152 to another device, such as the remote device 198.

[0126] FIG.16 depicts an example 1600 of a wearable electronic device 1602, illustrated as a "smart watch." In a particular aspect, the wearable electronic device 1602 includes, corresponds to, or is integrated within the device 102. The wearable electronic device 1602 includes a display screen 1604 and a camera 1612 (e.g., the image sensor 104). The ML model 140 is integrated in the wearable electronic device 1602, such as in the integrated circuit 1302. In a particular example, the wearable electronic device 1602 includes a haptic device that provides a haptic notification (e.g., vibrates) associated with display of image or video data that is based on image or video data that been captured by the camera 1612 and processed by the ML model 140, such as the output data 107, which may be displayed via the display screen 1604. For example, the haptic notification can cause a user to look at the wearable electronic device 1602 to watch video playback.

[0127] FIG.17 depicts an example 1700 in which the device 102 includes, corresponds to, or is integrated within a portable electronic device that corresponds to an extended reality device, such as augmented reality or mixed reality glasses 1702. The glasses 1702 include a holographic projection unit 1704 configured to project visual data onto a surface of a lens 1706 or to reflect the visual data off of a surface of the lens 1706 and onto the wearer's retina. The glasses 1702 include a camera 1712, such as the image sensor 104. The ML model 140 is integrated in the glasses 1702, such as in the integrated circuit 1302. In a particular example, the ML model 140 operates to perform image processing using a native aspect ratio at an autoregressive model. For example, the image data 105 may be received from the camera 1712, processed using the ML model 140, and the resulting output data 107 (e.g., an output image based on an output of the ML model 140) may be displayed via a projection onto the surface of the lens 1706 to enable display of images and / or video associated with augmented reality, mixed reality, or virtual reality scenes to the user while the glasses 1702 are worn.

[0128] FIG.18 depicts an example 1800 in which the device 102 includes, corresponds to, or is integrated within a portable electronic device that corresponds to aQUALCOMM Ref. No.2406633WO - 30 - virtual reality, augmented reality, or mixed reality headset 1802. The headset 1802 includes a camera 1812, such as the image sensor 104, and a visual display device 1804. The ML model 140 is integrated in the headset 1802, such as in the integrated circuit 1302. In a particular example, the ML model 140 operates to perform image processing using a native aspect ratio at an autoregressive model. For example, the image data 105 may be received from the camera 1812, processed using the ML model 140, and the resulting output data 107 (e.g., an output image based on an output of the ML model 140) may be displayed at the visual display device 1804 to enable display of images and / or video associated with augmented reality, mixed reality, or virtual reality scenes to the user while the headset 1802 is worn.

[0129] FIG.19 is an example 1900 of a wireless speaker and voice activated device 1902. In a particular aspect, the wireless speaker and voice activated device 1902 includes, corresponds to, or is integrated within the device 102. The wireless speaker and voice activated device 1902 can have wireless network connectivity and is configured to execute an assistant operation. The one or more processors 116 are included in the wireless speaker and voice activated device 1902 and include the ML model 140.

[0130] The wireless speaker and voice activated device 1902 includes a camera 1912, such as the image sensor 104, and a display device 1920. In a particular example, the ML model 140 operates to perform image processing using a native aspect ratio at an autoregressive model. For example, the image data 105 may be received from the camera 1912 and processed using the ML model 140, and the resulting output data 107 (e.g., an output image based on an output of the ML model 140) may be displayed at the display device 1920 and / or transmitted to a remote device, such as the remote device 198, for playback at the remote device.

[0131] In a particular aspect, the wireless speaker and voice activated device 1902 includes one or more microphones 1910 and one or more speakers 1904. During operation, in response to receiving a verbal command via the one or more microphones 1910, the wireless speaker and voice activated device 1902 can execute assistant operations, such as via execution of a voice activation system (e.g., an integratedQUALCOMM Ref. No.2406633WO - 31 - assistant application). The assistant operations can include adjusting a temperature, activating the camera 1912 to capture video or image content and displaying output image or video data based on the captured video content (e.g., the output data 107) at the display device 1920. In some examples, the assistant operations are performed responsive to receiving a command after a keyword or key phrase (e.g., “hello assistant”) received via the one or more microphones 1910.

[0132] FIG.20 depicts a first example 2000 in which the device 102 includes, corresponds to, or is integrated within a vehicle 2002, illustrated as a manned or unmanned aerial device (e.g., a package delivery drone). The ML model 140 is integrated in the vehicle 2002, such as in the integrated circuit 1302. The vehicle 2002 may also include a display device 2004 configured to display an output based on processing input data at the ML model 140, such as the output data 107.

[0133] In some implementations, the vehicle 2002 is manned (e.g., carries a pilot, one or more passengers, or both), the display device 2004 is internal to a cabin of the vehicle 2002, and the input data processing (e.g., image processing using a native aspect ratio at an autoregressive model) is performed using image and / or video capture via one or more cameras 2012. The input data processing may be used to generate navigational data, such as based in part on a segment map or classification corresponding to one or more objects or a scene in the proximity of the vehicle 2002, such as for playback to a pilot or a passenger of the vehicle 2002 and / or for semi-autonomous or autonomous operation of the vehicle 2002. In another implementation, the vehicle 2002 is unmanned, the input data processing (e.g., image processing using a native aspect ratio at an autoregressive model) is performed using image and / or video capture via the one or more cameras 2012 to generate navigational data corresponding to one or more objects or a scene in the proximity of the vehicle 2002, which may be displayed to a remote operator of the vehicle 2002 and / or used for semi-autonomous or autonomous operation of the vehicle 2002.

[0134] In some embodiments, the display device 2004 and the camera 2012 are mounted to an external surface of the vehicle 2002, and the input data processing at the ML model 140 is performed during video playback to one or more viewers external toQUALCOMM Ref. No.2406633WO - 32 - the vehicle 2002. For example, the vehicle 2002 may move (e.g., circle an outdoor audience during a concert) while playing out video or images based on video or image data captured via the camera 2012.

[0135] FIG.21 depicts a second example 2100 in which the device 102 includes, corresponds to, or is integrated within, a vehicle 2102, illustrated as a car. The ML model 140 is integrated in the vehicle 2102, such as in the integrated circuit 1302. In a particular example, the ML model 140 operates to perform input data processing based on image data received from one or more cameras 2112. The input data processing (e.g., image processing using a native aspect ratio at an autoregressive model) may be used to generate navigational data, such as based on image segmentation and / or classification corresponding to one or more objects or a scene in the proximity of the vehicle 2102, such as for playback of the navigational data to an operator of the vehicle 2102 via a display screen 2120 or a speaker 2110, and / or for semi-autonomous or autonomous operation of the vehicle 2102.

[0136] For example, in a particular embodiment, the vehicle 2102 may generate the image data 105 from the one or more cameras 2112, process the image data 105 at the ML model 140, and display the resulting output data 107 at the display screen 2120 of the vehicle 2102, store the resulting output data 107 and / or the model output data 152 at a memory of the vehicle 2102, and / or transmit the resulting output data 107 and / or the model output data 152 to another device, such as the remote device 198. In a particular embodiment, one or more of the cameras 2112 can be mounted to capture an interior scene including one or more other passengers of the vehicle 2102, such as to monitor children in a rear seat of the vehicle 2102. Additionally, or alternatively, one or more of the cameras 2112 can correspond to forward-facing cameras and / or rear-facing cameras that capture fields of view external to the vehicle 2102 in conjunction with autonomous or driver-assisted operation of the vehicle 2102.

[0137] FIG.22 illustrates an example of a method 2200 of input data processing. One or more operations of the method 2200 may be performed by at least one of the device 102, the one or more processors 116, or the system 100 of FIG.1, as an illustrative, non-limiting example.QUALCOMM Ref. No.2406633WO - 33 -

[0138] The method 2200 includes, at block 2202, obtaining an input image at a device. For example, the input image(s) 122 may be obtained from the input image source 120, such as via the image data 105 from the image sensor 104 or from the remote device 198.

[0139] The method 2200 includes, at block 2204, generating, at the device, input data that corresponds to a sequence of patches of the input image, the patches arranged to preserve an aspect ratio of the input image. For example, the device 102 processes the input image(s) 122 at the NAR-preserving patch generator 130 to generate the input data 136 including the sequence of patches 138 that preserve the NAR of the input image(s) 122.

[0140] The method 2200 includes, at block 2206, processing, at a trained transformer for next image patch prediction at the device, the input data to generate a transformer output corresponding to one or more predicted next patches of the sequence. For example, the trained transformer 142 processes the input data 136 to generate the transformer output 144 corresponding to one or more predicted next patches of the sequence of patches 138.

[0141] In some embodiments, the method 2200 includes including, in the input data, position information based on a horizontal image position and a vertical image position of each patch, where the position information corresponds to at least one of an absolute position embedding or a fractional position embedding, such as described with reference to the position information 542 added by the position adder 540 of the linear projection network 502 of FIG.5.

[0142] In some embodiments, the method 2200 includes obtaining training data that includes, for each training image of multiple training images, a respective sequence of patches of the training image, the patches arranged to preserve an aspect ratio of the training image, and performing a next-patch prediction training process on a vision transformer, using the training data, to generate the trained transformer, such as described with reference to the next-patch prediction training process 602 of FIG.6. According to an aspect, obtaining the training data includes, for each training image of the multiple training images, partitioning and scaling the training image to generateQUALCOMM Ref. No.2406633WO - 34 - multiple equally-sized patches for the training image that preserve the aspect ratio of the training image, such as described with respect to the operations 204 of FIG.2.

[0143] The method 2200 of FIG.22 may be implemented by a field-programmable gate array (FPGA) device, an application-specific integrated circuit (ASIC), a processing unit such as a central processing unit (CPU), a digital signal processor (DSP), a controller, another hardware device, firmware device, or any combination thereof. As an example, the method 2200 of FIG.22 may be performed by a processor that executes instructions, such as described with reference to FIG.23.

[0144] Referring to FIG.23, a block diagram of a particular illustrative implementation of a device is depicted and generally designated 2300. In various implementations, the device 2300 may have more or fewer components than illustrated in FIG.23. In an illustrative implementation, the device 2300 may correspond to the device 102 of FIG.1. In an illustrative implementation, the device 2300 may perform one or more operations described with reference to FIGS.1-22.

[0145] In a particular implementation, the device 2300 includes a processor 2306 (e.g., a CPU). The device 2300 may include one or more additional processors 2310 (e.g., one or more DSPs). In a particular implementation, the one or more processors 116 of FIG.1 correspond to the processor 2306, the processors 2310, or a combination thereof. For example, the processors 2310 may include the ML model 140. The ML model 140 may include one or more of the components of one or more of the examples of FIGS.1-6, or a combination thereof. The processors 2310 may also include a speech and music coder-decoder (CODEC) 2308. The speech and music CODEC 2308 may include a voice coder ("vocoder") encoder 2336, a vocoder decoder 2338, or a combination thereof.

[0146] In this context, the term “processor” refers to an integrated circuit consisting of logic cells, interconnects, input / output blocks, clock management components, memory, and optionally other special purpose hardware components, designed to execute instructions and perform various computational tasks. Examples of processors include, without limitation, CPUs, digital signal processors DSPs, neural processing units (NPUs), graphics processing units (GPUs), FPGAs, microcontrollers, quantumQUALCOMM Ref. No.2406633WO - 35 - processors, coprocessors, vector processors, other similar circuits, and variants and combinations thereof. In some cases, a processor can be integrated with other components, such as communication components, input / output components, etc. to form a system on a chip (SOC) device or a packaged electronic device.

[0147] Taking CPUs as a starting point, a CPU typically includes one or more processor cores, each of which includes a complex, interconnected network of transistors and other circuit components defining logic gates, memory elements, etc. A core is responsible for executing instructions to, for example, perform arithmetic and logical operations. Typically, a CPU includes an Arithmetic Logic Unit (ALU) that handles mathematical operations and a Control Unit that generates signals to coordinate the operation of other CPU components, such as to manage operations a fetch-decode- execute cycle.

[0148] CPUs and / or individual processor cores generally include local memory circuits, such as registers and cache to temporarily store data during operations. Registers include high-speed, small-sized memory units intimately connected to the logic cells of a CPU. Often registers include transistors arranged as groups of flip-flops, which are configured to store binary data. Caches include fast, on-chip memory circuits used to store frequently accessed data. Caches can be implemented, for example, using Static Random-Access Memory (SRAM) circuits.

[0149] Operations of a CPU (e.g., arithmetic operations, logic operations, and flow control operations) are directed by software and firmware. At the lowest level, the CPU includes an instruction set architecture (ISA) that specifies how individual operations are performed using hardware resources (e.g., registers, arithmetic units, etc.). Higher level software and firmware is translated into various combinations of ISA operations to cause the CPU to perform specific higher-level operations. For example, an ISA typically specifies how the hardware components of the CPU move and modify data to perform operations such as addition, multiplication, and subtraction, and high-level software is translated into sets of such operations to accomplish larger tasks, such as adding two columns in a spreadsheet. Generally, a CPU operates on various levels of software, including a kernel, an operating system, applications, and so forth, with eachQUALCOMM Ref. No.2406633WO - 36 - higher level of software generally being more abstracted from the ISA and usually more readily understandable by human users.

[0150] GPUs, NPUs, DSPs, microcontrollers, coprocessors, FPGAs, ASICS, and vector processors include components similar to those described above for CPUs. The differences among these various types of processors are generally related to the use of specialized interconnection schemes and ISAs to improve a processor’s ability to perform particular types of operations. For example, the logic gates, local memory circuits, and the interconnects therebetween of a GPU are specifically designed to improve parallel processing, sharing of data between processor cores, and vector operations, and the ISA of the GPU may define operations that take advantage of these structures. As another example, ASICs are highly specialized processors that include similar circuitry arranged and interconnected for a particular task, such as encryption or signal processing. As yet another example, FPGAs are programmable devices that include an array of configurable logic blocks (e.g., interconnect sets of transistors and memory elements) that can be configured (often on the fly) to perform customizable logic functions.

[0151] The device 2300 may include a memory 2386 and a CODEC 2334. The memory 2386 may include instructions 2356 that are executable by the one or more additional processors 2310 (or the processor 2306) to implement the functionality described with reference to the processor 116 of FIG.1. In a particular example, the memory 2386 corresponds to the memory 110 and the instructions 2356 correspond to the instructions 112 of FIG.1. The device 2300 may include the modem 118 coupled, via a transceiver 2350, to an antenna 2352. The device 2300 may also include one or more cameras 2394, one or more of which may correspond to the image sensor 104 of FIG.1.

[0152] The device 2300 may include a display 2328, such as the display device 106 of FIG.1, coupled to a display controller 2326. One or more speakers 2392, one or more microphones 2390, or a combination thereof, may be coupled to the CODEC 2334. The CODEC 2334 may include a digital-to-analog converter (DAC) 2302 and an analog-to- digital converter (ADC) 2304. In a particular implementation, the CODEC 2334 mayQUALCOMM Ref. No.2406633WO - 37 - receive analog signals from the microphones 2390, convert the analog signals to digital signals using the analog-to-digital converter 2304, and send the digital signals to the speech and music codec 2308. In a particular implementation, the speech and music codec 2308 may provide digital signals to the CODEC 2334. The CODEC 2334 may convert the digital signals to analog signals using the digital-to-analog converter 2302 and may provide the analog signals to the speakers 2392.

[0153] In a particular implementation, the device 2300 may be included in a system- in-package or system-on-chip device 2322. In a particular implementation, the memory 2386, the processor 2306, the processors 2310, the display controller 2326, the CODEC 2334, and the modem 118 are included in a system-in-package or system-on-chip device 2322. In a particular implementation, an input device 2330 (e.g., a keyboard, a touchscreen, or a pointing device) and a power supply 2344 are coupled to the system- in-package or system-on-chip device 2322. Moreover, in a particular implementation, as illustrated in FIG.23, the cameras 2394, the display 2328, the input device 2330, the speakers 2392, the microphones 2390, the antenna 2352, and the power supply 2344 are external to the system-in-package or system-on-chip device 2322. In a particular implementation, each of the cameras 2394, the display 2328, the input device 2330, the speakers 2392, the microphones 2390, the antenna 2352, and the power supply 2344 may be coupled to a component of the system-in-package or system-on-chip device 2322, such as an interface or a controller.

[0154] The device 2300 may include a smart speaker, a speaker bar, a mobile communication device, a smart phone, a cellular phone, a laptop computer, a computer, a tablet, a personal digital assistant, a display device, a television, a gaming console, a music player, a radio, a digital video player, a digital video disc (DVD) player, a tuner, a camera, a navigation device, a vehicle, a headset, an augmented reality headset, a mixed reality headset, a virtual reality headset, an aerial vehicle, a home automation system, a voice-activated device, a wireless speaker and voice activated device, a portable electronic device, a car, a vehicle, a computing device, a communication device, an internet-of-things (IoT) device, a virtual reality (VR) device, a base station, a mobile device, or any combination thereof.QUALCOMM Ref. No.2406633WO - 38 -

[0155] In conjunction with the described techniques, an apparatus includes means for obtaining an input image. In an example, the means for obtaining the input image can include the input image source 120, the image sensor 104, the modem 118, the one or more processors 116, the device 102, the system 100, one or more other circuits or devices to obtain an input image, or a combination thereof.

[0156] The apparatus also includes means for generating input data that corresponds to a sequence of patches of the input image, the patches arranged to preserve an aspect ratio of the input image. In an example, the means for generating the input data can include the NAR-preserving patch generator 130, the one or more processors 116, the device 102, the system 100, the linear projector 520, the position adder 540, the linear projection network 502, one or more other circuits or devices configured to generate input data that corresponds to a sequence of patches of the input image, the patches arranged to preserve an aspect ratio of the input image, or a combination thereof.

[0157] The apparatus also includes means for processing, at a trained transformer for next image patch prediction, the input data to generate a transformer output corresponding to one or more predicted next patches of the sequence. In an example, the means for processing the input data to generate a transformer output corresponding to one or more predicted next patches of the sequence can include the one or more processors 116, the trained transformer 142 executed by the one or more processors 116, the ML model 140 executed by the one or more processors 116, the device 102, the system 100, one or more other circuits or devices configured to process, at a trained transformer for next image patch prediction, the input data to generate a transformer output corresponding to one or more predicted next patches of the sequence, or a combination thereof.

[0158] In some implementations, a non-transitory computer-readable medium (e.g., a computer-readable storage device, such as the memory 110) includes instructions (e.g., the instructions 112) that, when executed by one or more processors (e.g., the one or more processors 116), cause the one or more processors to perform operations corresponding to at least a portion of any of the techniques described with reference to FIGS.1-23, including the method 2200 of FIG.22, or any combination thereof. In anQUALCOMM Ref. No.2406633WO - 39 - example, the instructions, when executed by the one or more processors, cause the one or more processors to obtain an input image (e.g., the input image(s) 122). The instructions, when executed by the one or more processors, cause the one or more processors to generate input data (e.g., the input data 136) that corresponds to a sequence of patches (e.g., the sequence of patches 138) of the input image, the patches arranged to preserve an aspect ratio of the input image. The instructions, when executed by the one or more processors, also cause the one or more processors to process, at a trained transformer for next image patch prediction (e.g., the trained transformer 142), the input data to generate a transformer output (e.g., the transformer output 144) corresponding to one or more predicted next patches of the sequence.

[0159] Particular aspects of the disclosure are described below in the following sets of interrelated Examples:

[0160] According to Example 1, a device includes a memory configured to store data corresponding to a trained transformer for next image patch prediction; and one or more processors, coupled to the memory, wherein the one or more processors are configured to obtain an input image; generate input data that corresponds to a sequence of patches of the input image, the patches arranged to preserve an aspect ratio of the input image; and process the input data at the trained transformer to generate a transformer output that corresponds to one or more predicted next patches of the sequence.

[0161] Example 2 includes the device of Example 1, wherein the trained transformer corresponds to an autoregressive vision transformer.

[0162] Example 3 includes the device of Example 1 or Example 2, wherein the one or more processors are configured to process the transformer output at a prediction head.

[0163] Example 4 includes the device of any of Examples 1 to 3, wherein the one or more processors are configured to include, in the input data, position information based on a horizontal image position and a vertical image position of each patch.QUALCOMM Ref. No.2406633WO - 40 -

[0164] Example 5 includes the device of Example 4, wherein the position information corresponds to an absolute position embedding.

[0165] Example 6 includes the device of Example 4, wherein the position information corresponds to a fractional position embedding.

[0166] Example 7 includes the device of any of Examples 4 to 6, wherein the one or more processors are configured to add the position information at a linear projection network that is coupled to an input to the trained transformer.

[0167] Example 8 includes the device of any of Examples 1 to 7, wherein the one or more processors are configured to obtain training data that includes, for each training image of multiple training images, a respective sequence of patches of the training image, the patches arranged to preserve an aspect ratio of the training image; and perform a next-patch prediction training process on a vision transformer, using the training data, to generate the trained transformer.

[0168] Example 9 includes the device of Example 8, wherein the one or more processors are configured to, for each training image of the multiple training images, partition and scale the training image to generate multiple equally-sized patches for the training image that preserve the aspect ratio of the training image.

[0169] Example 10 includes the device of Example 8 or Example 9, wherein, during the next-patch prediction training process, the one or more processors are configured to, for each training image of the multiple training images: process an output of the vision transformer at a pre-training head to generate predicted next patches in the sequence; and update the vision transformer based on accuracy of the predicted next patches.

[0170] Example 11 includes the device of any of Examples 1 to 10 and further includes one or more cameras coupled to the one or more processors, the one or more cameras configured to generate the input image.

[0171] Example 12 includes the device of any of Examples 1 to 11 and further includes a modem coupled to the one or more processors, the modem configured to receive the input image from a second device.QUALCOMM Ref. No.2406633WO - 41 -

[0172] Example 13 includes the device of any of Examples 1 to 12, wherein the one or more processors are integrated in at least one of a mobile phone, a tablet computer device, a wearable electronic device, a virtual reality headset, a mixed reality headset, an augmented reality headset, or a camera device.

[0173] Example 14 includes the device of any of Examples 1 to 12, wherein the one or more processors are integrated in a vehicle, the vehicle further including one or more cameras configured to generate the input image.

[0174] Example 15 includes the device of any of Examples 1 to 14, wherein the one or more processors are included in an integrated circuit.

[0175] According to Example 16, a method includes obtaining, at a device, an input image; generating, at the device, input data that corresponds to a sequence of patches of the input image, the patches arranged to preserve an aspect ratio of the input image; and processing, at a trained transformer for next image patch prediction at the device, the input data to generate a transformer output corresponding to one or more predicted next patches of the sequence.

[0176] Example 17 includes the method of Example 16, wherein the trained transformer corresponds to an autoregressive vision transformer.

[0177] Example 18 includes the method of Example 16 or Example 17, and further includes processing the transformer output at a prediction head.

[0178] Example 19 includes the method of any of Examples 16 to 18, and further includes including, in the input data, position information based on a horizontal image position and a vertical image position of each patch, wherein the position information corresponds to at least one of an absolute position embedding or a fractional position embedding.

[0179] Example 20 includes the method of any of Examples 16 to 18, and further includes including, in the input data, position information based on a horizontal image position and a vertical image position of each patch.QUALCOMM Ref. No.2406633WO - 42 -

[0180] Example 21 includes the method of Example 20, wherein the position information corresponds to an absolute position embedding.

[0181] Example 22 includes the method of Example 20, wherein the position information corresponds to a fractional position embedding.

[0182] Example 23 includes the method of any of Examples 16 to 22, and further includes obtaining training data that includes, for each training image of multiple training images, a respective sequence of patches of the training image, the patches arranged to preserve an aspect ratio of the training image; and performing a next-patch prediction training process on a vision transformer, using the training data, to generate the trained transformer.

[0183] Example 24 includes the method of Example 23, wherein obtaining the training data includes, for each training image of the multiple training images, partitioning and scaling the training image to generate multiple equally-sized patches for the training image that preserve the aspect ratio of the training image.

[0184] Example 25 includes the method of Example 23 or Example 24, wherein the next-patch prediction training process includes, for each training image of the multiple training images: processing an output of the vision transformer at a pre-training head to generate predicted next patches in the sequence; and updating the vision transformer based on accuracy of the predicted next patches.

[0185] Example 26 includes the method of any of Examples 16 to 25 and further includes generating the input image at one or more cameras.

[0186] Example 27 includes the method of any of Examples 16 to 26 and further includes receiving the input image from a second device via a modem.

[0187] Example 28 includes the method of any of Examples 16 to 27, wherein the device is integrated in at least one of a mobile phone, a tablet computer device, a wearable electronic device, a virtual reality headset, a mixed reality headset, an augmented reality headset, or a camera device.QUALCOMM Ref. No.2406633WO - 43 -

[0188] Example 29 includes the method of any of Examples 16 to 27, wherein the device is integrated in a vehicle that includes one or more cameras configured to generate the input image.

[0189] Example 30 includes the method of any of Examples 16 to 27, wherein the device is included in an integrated circuit.

[0190] According to Example 31, a device includes: a memory configured to store instructions; and a processor configured to execute the instructions to perform the method of any of Examples 16 to 30.

[0191] According to Example 32, a non-transitory computer-readable medium stores instructions that, when executed by a processor, cause the processor to perform the method of any of Examples 16 to 30.

[0192] According to Example 33, an apparatus includes means for carrying out the method of any of Examples 16 to 30.

[0193] According to Example 34, a non-transitory computer-readable medium includes instructions that, when executed by one or more processors, cause the one or more processors to obtain an input image; generate input data that corresponds to a sequence of patches of the input image, the patches arranged to preserve an aspect ratio of the input image; and process, at a trained transformer for next image patch prediction, the input data to generate a transformer output corresponding to one or more predicted next patches of the sequence.

[0194] Those of skill would further appreciate that the various illustrative logical blocks, configurations, circuits, and algorithm steps described in connection with the implementations disclosed herein may be implemented as electronic hardware, computer software executed by a processing device such as a hardware processor, or combinations of both. Various illustrative components, blocks, configurations, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or executable software depends upon the particular application and design constraints imposed on the overall system. SkilledQUALCOMM Ref. No.2406633WO - 44 - artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.

[0195] The steps of a method or algorithm described in connection with the implementations disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in a memory device, such as random access memory (RAM), magnetoresistive random access memory (MRAM), spin-torque transfer MRAM (STT- MRAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disk, a removable disk, a compact disc read-only memory (CD-ROM), or any other form of non-transient storage medium known in the art. An exemplary memory device is coupled to the processor such that the processor can read data from, and write data to, the memory device. In the alternative, the memory device may be integral to the processor. The processor and the storage medium may reside in an application-specific integrated circuit (ASIC). The ASIC may reside in a computing device or a user terminal. In the alternative, the processor and the storage medium may reside as discrete components in a computing device or a user terminal.

[0196] The previous description of the disclosed implementations is provided to enable a person skilled in the art to make or use the disclosed implementations. Various modifications to these implementations will be readily apparent to those skilled in the art, and the principles defined herein may be applied to other implementations without departing from the scope of the disclosure. Thus, the present disclosure is not intended to be limited to the implementations shown herein but is to be accorded the widest scope possible consistent with the principles and novel features as defined by the following claims.

Claims

QUALCOMM Ref. No.2406633WO - 45 - WHAT IS CLAIMED IS:

1. A device comprising: a memory configured to store data corresponding to a trained transformer for next image patch prediction; and one or more processors, coupled to the memory, wherein the one or more processors are configured to: obtain an input image; generate input data that corresponds to a sequence of patches of the input image, the patches arranged to preserve an aspect ratio of the input image; and process the input data at the trained transformer to generate a transformer output that corresponds to one or more predicted next patches of the sequence.

2. The device of claim 1, wherein the trained transformer corresponds to an autoregressive vision transformer.

3. The device of claim 1, wherein the one or more processors are configured to process the transformer output at a prediction head.

4. The device of claim 1, wherein the one or more processors are configured to include, in the input data, position information based on a horizontal image position and a vertical image position of each patch.

5. The device of claim 4, wherein the position information corresponds to an absolute position embedding.

6. The device of claim 4, wherein the position information corresponds to a fractional position embedding.

7. The device of claim 4, wherein the one or more processors are configured to add the position information at a linear projection network that is coupled to an input toQUALCOMM Ref. No.2406633WO - 46 - the trained transformer.

8. The device of claim 1, wherein the one or more processors are configured to: obtain training data that includes, for each training image of multiple training images, a respective sequence of patches of the training image, the patches arranged to preserve an aspect ratio of the training image; and perform a next-patch prediction training process on a vision transformer, using the training data, to generate the trained transformer.

9. The device of claim 8, wherein the one or more processors are configured to, for each training image of the multiple training images, partition and scale the training image to generate multiple equally-sized patches for the training image that preserve the aspect ratio of the training image.

10. The device of claim 8, wherein, during the next-patch prediction training process, the one or more processors are configured to, for each training image of the multiple training images: process an output of the vision transformer at a pre-training head to generate predicted next patches in the sequence; and update the vision transformer based on accuracy of the predicted next patches.

11. The device of claim 1, further comprising one or more cameras coupled to the one or more processors, the one or more cameras configured to generate the input image.

12. The device of claim 1, further comprising a modem coupled to the one or more processors, the modem configured to receive the input image from a second device.

13. The device of claim 1, wherein the one or more processors are integrated in at least one of a mobile phone, a tablet computer device, a wearable electronic device, a virtual reality headset, a mixed reality headset, an augmented reality headset, or a camera device.

14. The device of claim 1, wherein the one or more processors are integrated in aQUALCOMM Ref. No.2406633WO - 47 - vehicle, the vehicle further including one or more cameras configured to generate the input image.

15. The device of claim 1, wherein the one or more processors are included in an integrated circuit.

16. A method comprising: obtaining, at a device, an input image; generating, at the device, input data that corresponds to a sequence of patches of the input image, the patches arranged to preserve an aspect ratio of the input image; and processing, at a trained transformer for next image patch prediction at the device, the input data to generate a transformer output corresponding to one or more predicted next patches of the sequence.

17. The method of claim 16, further comprising including, in the input data, position information based on a horizontal image position and a vertical image position of each patch, wherein the position information corresponds to at least one of an absolute position embedding or a fractional position embedding.

18. The method of claim 16, further comprising: obtaining training data that includes, for each training image of multiple training images, a respective sequence of patches of the training image, the patches arranged to preserve an aspect ratio of the training image; and performing a next-patch prediction training process on a vision transformer, using the training data, to generate the trained transformer.

19. The method of claim 18, wherein obtaining the training data includes, for each training image of the multiple training images, partitioning and scaling the training image to generate multiple equally-sized patches for the training image that preserve the aspect ratio of the training image.

20. A non-transitory computer-readable medium comprising instructions that,QUALCOMM Ref. No.2406633WO - 48 - when executed by one or more processors, cause the one or more processors to: obtain an input image; generate input data that corresponds to a sequence of patches of the input image, the patches arranged to preserve an aspect ratio of the input image; and process, at a trained transformer for next image patch prediction, the input data to generate a transformer output corresponding to one or more predicted next patches of the sequence.

Images